← All Webinars | L.A.B.S. #9

Level: Beginner🐣 

Experience a live AI survey tool demo, learn strategies to ensure data quality, and explore how AI can optimize survey results.

+ 

1
00:00:10.265 --> 00:00:11.225
Hello everyone.

2
00:00:11.575 --> 00:00:12.585
Welcome, welcome.

3
00:00:12.995 --> 00:00:15.865
We're gonna give it another minute before we get started.

4
00:00:16.705 --> 00:00:17.645
So thank you for joining.

5
00:00:24.225 --> 00:00:27.665
I am looking forward to this discussion, this four,

6
00:00:28.085 --> 00:00:30.625
our fourth part of our five part series.

7
00:00:32.255 --> 00:00:33.805
We're gonna get started in a minute.

8
00:00:55.865 --> 00:00:59.565
All right, well, excellent. Excellent.

9
00:01:00.405 --> 00:01:01.695
Well, hi everyone.

10
00:01:02.105 --> 00:01:04.735
Thank you so much for joining our webinar today.

11
00:01:05.525 --> 00:01:07.945
My name is Brodie Wise, I'm the Executive Vice President

12
00:01:07.945 --> 00:01:10.425
of Business Development and Marketing here at ITS.

13
00:01:11.015 --> 00:01:13.065
I'll be your host and moderator today.

14
00:01:14.755 --> 00:01:18.285
Once again, thank you for joining this fourth session

15
00:01:18.545 --> 00:01:22.285
of our Summer, Summer Demo series, AI in Practice.

16
00:01:23.175 --> 00:01:25.795
So just a few things for admin, just wanna share,

17
00:01:26.255 --> 00:01:28.235
if you do have a question, there's a Q

18
00:01:28.235 --> 00:01:30.635
and A section, post your questions there,

19
00:01:31.255 --> 00:01:33.995
and based on the number of clicks, you can rank

20
00:01:34.055 --> 00:01:36.315
and prioritize things that you have questions about.

21
00:01:36.655 --> 00:01:39.315
And then we will try to address 'em through the session.

22
00:01:40.685 --> 00:01:42.565
I, if there are too many questions,

23
00:01:42.795 --> 00:01:45.085
what we'll do is we'll answer 'em post webinar.

24
00:01:46.865 --> 00:01:48.005
If, for those of you

25
00:01:48.005 --> 00:01:50.525
who just joined the webinar will be recorded

26
00:01:51.065 --> 00:01:52.925
and we will share a link on

27
00:01:52.925 --> 00:01:55.285
that landing page that we have set up.

28
00:01:56.105 --> 00:01:59.165
So before you exit from the webinar today,

29
00:01:59.455 --> 00:02:01.685
which will wrap up at 12:45,

30
00:02:02.455 --> 00:02:04.325
there will be a survey at the end

31
00:02:04.505 --> 00:02:06.485
and it will, so don't click out too quickly,

32
00:02:07.015 --> 00:02:08.805
it'll at the end through the Zoom window.

33
00:02:09.385 --> 00:02:12.285
We really want your feedback about this, the webinar

34
00:02:12.585 --> 00:02:14.805
and maybe suggestions and some ideas.

35
00:02:15.445 --> 00:02:16.565
'cause we're always trying to do it better

36
00:02:16.665 --> 00:02:18.365
and make sure we're giving you great

37
00:02:18.365 --> 00:02:19.765
content that you find useful.

38
00:02:20.825 --> 00:02:22.485
So now less of me,

39
00:02:22.965 --> 00:02:25.005
'cause you guys didn't come to listen to me today.

40
00:02:25.385 --> 00:02:28.605
Let me turn over the next 40 minutes to Jeffrey Spranza

41
00:02:28.705 --> 00:02:31.805
and Craig Boas to lead our Part Four session,

42
00:02:32.315 --> 00:02:35.885
Enhance Data Analysis and Optimize Survey Results.

43
00:02:36.395 --> 00:02:37.965
Welcome Jeffrey and Craig.

44
00:02:39.445 --> 00:02:44.065
Thanks Brodie. Uh, so welcome everybody and,

45
00:02:44.125 --> 00:02:45.185
and thank you all for coming.

46
00:02:45.525 --> 00:02:48.505
And, uh, today what we're gonna do is we're gonna give you a

47
00:02:48.555 --> 00:02:52.025
sneak peek at the upcoming release of our new survey tool,

48
00:02:52.405 --> 00:02:54.825
uh, which includes an AI analysis function.

49
00:02:55.125 --> 00:02:57.505
Uh, we'll talk at first a little bit about why we created

50
00:02:57.565 --> 00:03:00.325
the tool, but then I'll jump into demonstrating the AI

51
00:03:00.325 --> 00:03:02.685
feature, which you'll get to try live in this session.

52
00:03:03.305 --> 00:03:04.885
And then we'll take a look behind the curtain

53
00:03:05.065 --> 00:03:06.165
and we'll go through a little bit

54
00:03:06.165 --> 00:03:07.565
of the product process we went

55
00:03:07.565 --> 00:03:09.645
through when we were incorporating the

56
00:03:09.785 --> 00:03:11.005
AI feature specifically.

57
00:03:11.385 --> 00:03:13.605
Uh, Jeffrey will ask me some questions as we go just

58
00:03:13.605 --> 00:03:14.805
to keep the conversation moving.

59
00:03:15.145 --> 00:03:17.245
But as Brodie said, please do feel free

60
00:03:17.245 --> 00:03:19.045
to put your own questions into the chat

61
00:03:19.465 --> 00:03:21.205
and I'll try my best to respond

62
00:03:21.205 --> 00:03:22.725
to whatever sparks your curiosity.

63
00:03:23.665 --> 00:03:25.005
Um, so Jeffrey, you wanna,

64
00:03:25.235 --> 00:03:26.485
Yeah, so thanks Craig.

65
00:03:26.665 --> 00:03:28.885
Uh, so you know, Craig, you

66
00:03:28.885 --> 00:03:30.725
and I have had this conversation many times over

67
00:03:30.745 --> 00:03:32.845
and some of these questions that I'm gonna ask you today are

68
00:03:32.845 --> 00:03:34.285
based upon the conversations we've had

69
00:03:34.425 --> 00:03:36.765
as we were developing, uh, survey.

70
00:03:36.905 --> 00:03:39.205
So can you talk about why we decided

71
00:03:39.265 --> 00:03:42.645
to build survey functionality into our software when there

72
00:03:42.645 --> 00:03:44.005
are plenty of third party options

73
00:03:44.005 --> 00:03:45.045
that were already available?

74
00:03:45.505 --> 00:03:47.765
Uh, many of 'em, some of our clients use today,

75
00:03:48.645 --> 00:03:49.645
Right? And, and we have known

76
00:03:49.645 --> 00:03:50.205
for a while

77
00:03:50.235 --> 00:03:53.045
that our customers are using third party survey products

78
00:03:53.045 --> 00:03:56.005
to gather information from candidates from subject

79
00:03:56.005 --> 00:03:57.325
matter experts and others.

80
00:03:57.785 --> 00:04:00.245
And we realized we saw some opportunities, uh,

81
00:04:00.245 --> 00:04:01.805
for bringing the surveys in-house

82
00:04:02.185 --> 00:04:04.765
and ways we could take advantage of some synergies

83
00:04:04.765 --> 00:04:07.285
with our software to make those surveys

84
00:04:07.285 --> 00:04:08.405
even more useful for them.

85
00:04:09.465 --> 00:04:12.565
So, you know, when we're talking about surveys, what type

86
00:04:12.565 --> 00:04:15.445
of surveys do we see programs sending

87
00:04:15.505 --> 00:04:17.285
to users, uh, for example?

88
00:04:17.875 --> 00:04:20.965
Well, um, it's important to note that like as you said,

89
00:04:20.995 --> 00:04:23.165
surveys are not restricted just to candidates.

90
00:04:23.165 --> 00:04:26.325
They can send them to subject matter experts as part

91
00:04:26.325 --> 00:04:28.845
of a standard setting or a job task analysis.

92
00:04:29.155 --> 00:04:31.765
Some programs might choose to survey administrators

93
00:04:31.765 --> 00:04:34.405
and test to get feedback about their experience

94
00:04:34.405 --> 00:04:35.445
administrating the test.

95
00:04:35.985 --> 00:04:37.845
Uh, the tool doesn't impose any limits.

96
00:04:38.035 --> 00:04:39.325
There's nothing built into it

97
00:04:39.325 --> 00:04:40.685
that restricts it to candidates.

98
00:04:40.985 --> 00:04:42.205
It just has some features

99
00:04:42.205 --> 00:04:43.965
that make it particularly well-suited

100
00:04:43.965 --> 00:04:45.045
for surveying candidates.

101
00:04:45.585 --> 00:04:46.925
Hey, could you give us some examples?

102
00:04:47.005 --> 00:04:49.565
I mean, I think it would be really helpful to, to send,

103
00:04:49.825 --> 00:04:51.365
you know, cement what we're doing here.

104
00:04:51.915 --> 00:04:53.765
Sure. Uh, so some of them are related.

105
00:04:53.765 --> 00:04:56.045
Some of those features are related to survey delivery.

106
00:04:56.465 --> 00:04:58.005
Uh, since we're delivering the survey,

107
00:04:58.105 --> 00:04:59.325
we can control the interface,

108
00:04:59.515 --> 00:05:01.445
make it look like a continuation of the test,

109
00:05:01.455 --> 00:05:04.165
which just feels more cohesive in, in terms

110
00:05:04.165 --> 00:05:06.485
of the experience that the person has taking the survey.

111
00:05:07.145 --> 00:05:09.245
Uh, more importantly, it, uh,

112
00:05:09.355 --> 00:05:11.765
lets us access our event system.

113
00:05:12.265 --> 00:05:15.735
So, uh, for example, um, well

114
00:05:15.735 --> 00:05:18.655
before I get into that, we have, uh, we can link surveys

115
00:05:18.655 --> 00:05:20.575
to events and data already stored in our system.

116
00:05:20.955 --> 00:05:22.695
Uh, for example, we frequently have a

117
00:05:22.695 --> 00:05:23.735
candidate's email address.

118
00:05:23.875 --> 00:05:26.495
So if the customer doesn't want to take a survey, uh,

119
00:05:26.495 --> 00:05:29.335
doesn't want the candidate to take the survey immediately at

120
00:05:29.335 --> 00:05:31.655
the end of a test, we can configure it to go out

121
00:05:31.655 --> 00:05:34.615
by email either right away or after a custom delay.

122
00:05:34.995 --> 00:05:36.695
And they don't have to be just test events.

123
00:05:37.035 --> 00:05:38.935
Uh, there are events that we use, for example,

124
00:05:39.115 --> 00:05:41.535
in our continuous learning model, which we're using

125
00:05:41.565 --> 00:05:42.895
with a lot of medical boards

126
00:05:42.895 --> 00:05:45.615
and involves questions delivered over time instead

127
00:05:45.615 --> 00:05:46.855
of in one big test.

128
00:05:47.315 --> 00:05:48.535
And those events have,

129
00:05:48.535 --> 00:05:51.495
those have events triggered when candidates pass all sorts

130
00:05:51.495 --> 00:05:53.535
of milestones, uh, when they create an account,

131
00:05:53.535 --> 00:05:54.615
when they finish a group of

132
00:05:54.815 --> 00:05:56.095
questions, all those kinds of things.

133
00:05:56.675 --> 00:05:59.365
And what's really cool is since these events are per

134
00:05:59.365 --> 00:06:01.565
candidate, customers can target those

135
00:06:01.565 --> 00:06:02.725
surveys very precisely.

136
00:06:02.985 --> 00:06:04.605
Uh, instead of just mass mailing a list

137
00:06:04.605 --> 00:06:07.005
of all your candidates, although that's also an option,

138
00:06:07.355 --> 00:06:08.805
they can configure their surveys

139
00:06:08.805 --> 00:06:11.365
to reach people exactly when the survey becomes relevant

140
00:06:11.495 --> 00:06:13.925
based upon the candidate's interactions with our system.

141
00:06:14.835 --> 00:06:17.335
Uh, there's also some advantages on the analysis side.

142
00:06:17.675 --> 00:06:20.415
Uh, for example, we already have information about the

143
00:06:20.415 --> 00:06:22.815
candidate in our system so we can link their survey

144
00:06:23.215 --> 00:06:24.375
responses to that candidate

145
00:06:24.375 --> 00:06:26.855
or even to a test result without relying on them

146
00:06:27.035 --> 00:06:28.615
to self-identify in the survey.

147
00:06:29.115 --> 00:06:31.215
So if the customer wants to, for example,

148
00:06:31.285 --> 00:06:33.535
look at correlations between test performance

149
00:06:33.755 --> 00:06:35.175
and their response to a question

150
00:06:35.175 --> 00:06:36.375
on the survey, they can do that.

151
00:06:37.235 --> 00:06:38.975
And while this isn't an exhaustive list,

152
00:06:38.975 --> 00:06:40.895
we're not really here to talk about all the ways

153
00:06:40.895 --> 00:06:42.655
that the survey component works.

154
00:06:43.205 --> 00:06:45.575
It's enough, I guess, to know that we have this component.

155
00:06:45.595 --> 00:06:48.095
And one of the things that allows us to do is

156
00:06:48.115 --> 00:06:51.055
to add things like our SparkAI technology

157
00:06:51.235 --> 00:06:53.295
to help analyze survey responses

158
00:06:53.405 --> 00:06:55.695
because we get to control the pace of innovation.

159
00:06:56.845 --> 00:06:59.225
All right, so we've been talking about surveys generally,

160
00:06:59.885 --> 00:07:02.465
but your last point is why we're here today, right?

161
00:07:02.605 --> 00:07:05.545
So can you tell us more about how we are using SparkAI

162
00:07:05.605 --> 00:07:07.265
to analyze their survey responses?

163
00:07:07.475 --> 00:07:09.105
Let's, let's go dig deeper there.

164
00:07:09.695 --> 00:07:13.425
Yeah, sure. Um, so surveys are great for getting feedback,

165
00:07:13.685 --> 00:07:15.945
but when you get your feedback, you need a way to turn

166
00:07:15.945 --> 00:07:18.105
that into actionable information.

167
00:07:18.605 --> 00:07:20.905
And for certain types of questions like multiple choice,

168
00:07:21.255 --> 00:07:22.385
this is a solved problem.

169
00:07:23.125 --> 00:07:25.065
We have bar graphs and pie charts

170
00:07:25.065 --> 00:07:26.985
and all sorts of things that work really well

171
00:07:26.985 --> 00:07:29.225
for summarizing those kinds of responses.

172
00:07:29.405 --> 00:07:31.145
And we're not gonna talk about those today.

173
00:07:31.145 --> 00:07:32.265
They're not that interesting.

174
00:07:32.265 --> 00:07:35.375
People know how to use them, but sometimes the information

175
00:07:35.375 --> 00:07:37.535
we want to elicit isn't really possible to gather

176
00:07:37.565 --> 00:07:38.935
with a multiple choice question.

177
00:07:39.385 --> 00:07:41.655
Maybe we don't know all the possible answers in advance.

178
00:07:41.655 --> 00:07:43.135
Maybe there's too many to list.

179
00:07:43.465 --> 00:07:45.695
Maybe we think suggesting answers will limit

180
00:07:45.695 --> 00:07:47.055
what survey respondents consider.

181
00:07:47.475 --> 00:07:48.575
And those are just some

182
00:07:48.575 --> 00:07:51.615
of the situations when really an open-ended text response

183
00:07:51.895 --> 00:07:53.855
question is the right tool for the job,

184
00:07:54.755 --> 00:07:56.375
but they come with a downside.

185
00:07:56.375 --> 00:07:57.695
And that's what got us thinking.

186
00:07:58.195 --> 00:08:01.575
You get this massive text responses and it's challenging

187
00:08:01.575 --> 00:08:04.535
and it's time consuming to boil all of those responses down

188
00:08:05.085 --> 00:08:06.815
into something that you can really use.

189
00:08:07.595 --> 00:08:09.055
And that's where AI and,

190
00:08:09.075 --> 00:08:12.215
and I'm gonna use the term AI here, uh, for clarity,

191
00:08:12.245 --> 00:08:14.655
what I'm talking about here is large language models

192
00:08:14.655 --> 00:08:17.255
because there are other types of ai, neural networks

193
00:08:17.255 --> 00:08:20.415
and things like that, but large language models specifically

194
00:08:20.475 --> 00:08:22.055
are really well suited to the task.

195
00:08:22.475 --> 00:08:23.935
And that's why we chose that.

196
00:08:24.715 --> 00:08:27.135
And I could talk a lot about it,

197
00:08:27.155 --> 00:08:29.295
but I think the most effective way to show everyone

198
00:08:29.315 --> 00:08:31.695
how this works is with a demonstration.

199
00:08:31.795 --> 00:08:34.295
So in the spirit of our demo days theme, um,

200
00:08:34.715 --> 00:08:38.095
I'm gonna ask Brodie, could you please post a link into

201
00:08:38.095 --> 00:08:39.175
the chat to a survey?

202
00:08:40.605 --> 00:08:42.145
And, and as he's doing that, Craig,

203
00:08:42.465 --> 00:08:45.825
I remember doing surveys and I'd get a paper copy

204
00:08:45.965 --> 00:08:46.985
and people would be written,

205
00:08:47.415 --> 00:08:49.025
have written in all the responses,

206
00:08:49.245 --> 00:08:50.865
and I have to review each one of those.

207
00:08:51.045 --> 00:08:53.385
And then I'd go and highlight or underline

208
00:08:53.685 --> 00:08:57.485
or try to keep notes of what I saw commonly and,

209
00:08:57.585 --> 00:08:58.845
and kind of boil that down.

210
00:08:59.385 --> 00:09:01.405
And so, uh, I'm thinking back

211
00:09:01.405 --> 00:09:02.925
to all those surveys thinking this would

212
00:09:02.925 --> 00:09:04.005
be a great feature to have.

213
00:09:05.145 --> 00:09:09.015
Great. Greg, I posted, I posted the link

214
00:09:09.035 --> 00:09:10.215
for you in the chat for everyone.

215
00:09:10.405 --> 00:09:12.975
Alright, thanks. So if, if everyone would be

216
00:09:13.235 --> 00:09:14.815
so kind is to click the link.

217
00:09:15.235 --> 00:09:18.775
Um, when you're doing it, uh, I'd particularly like you

218
00:09:18.775 --> 00:09:20.575
to pay attention to the last two questions,

219
00:09:20.575 --> 00:09:23.255
which are the ones that ask for a text response.

220
00:09:23.595 --> 00:09:26.695
And the kind of interactive part of this demo is that,

221
00:09:26.715 --> 00:09:28.975
you know, look at them, think about them, answer them.

222
00:09:29.075 --> 00:09:30.935
If they don't really apply to you, make something up.

223
00:09:31.115 --> 00:09:34.095
But just put in some text in there that makes sense.

224
00:09:34.595 --> 00:09:39.375
And the idea is that we're then going to kind of together

225
00:09:40.035 --> 00:09:43.375
run those answers through an AI analysis feature

226
00:09:43.755 --> 00:09:46.135
and I'll show you how it turns a massive text.

227
00:09:46.235 --> 00:09:49.095
And how many people do we have in the chat today? Uh oh,

228
00:09:49.095 --> 00:09:50.655
That link is not working, Craig.

229
00:09:51.635 --> 00:09:54.695
So, um, I noticed that, 'cause I clicked on the link too.

230
00:09:54.795 --> 00:09:58.095
And we have a couple of, uh, people on the, uh, call

231
00:09:58.115 --> 00:10:00.535
who have also noted that, uh, if we could check

232
00:10:00.535 --> 00:10:02.015
that, that you are on.

233
00:10:02.015 --> 00:10:04.055
I'm gonna give you a moment to, to do. So.

234
00:10:04.215 --> 00:10:06.095
I, and I am seeing it.

235
00:10:07.235 --> 00:10:10.805
That is because, uh,

236
00:10:12.285 --> 00:10:13.895
that is the, the link

237
00:10:13.895 --> 00:10:16.175
that you sent is not the link that Brodie posted.

238
00:10:16.435 --> 00:10:18.855
So Brodie, if you could, uh, copy

239
00:10:18.855 --> 00:10:20.335
and paste the one out of the earlier.

240
00:10:23.965 --> 00:10:25.905
Is it different? Yes, it is.

241
00:10:26.255 --> 00:10:27.625
Yeah, it is. We

242
00:10:34.835 --> 00:10:36.555
are experiencing technical difficulties.

243
00:10:37.355 --> 00:10:40.215
I i, I know what happened there.

244
00:10:40.285 --> 00:10:43.385
It's a is a Zoom thing.

245
00:10:52.125 --> 00:10:54.145
Birdie, I think you're, you're speaking but you're muted.

246
00:10:55.025 --> 00:10:57.685
Can you copy it from, um, post it in teams for me

247
00:10:57.685 --> 00:10:58.925
and I will copy it from there.

248
00:10:59.045 --> 00:11:00.165
I cannot copy it from

249
00:11:00.745 --> 00:11:02.365
Oh yeah, The Zoom link.

250
00:11:02.795 --> 00:11:06.565
Yeah, zoom will click the link, not copy the link.

251
00:11:07.955 --> 00:11:09.855
All right. Sorry everybody just taking a moment,

252
00:11:16.055 --> 00:11:18.425
Just one more minute and we'll get

253
00:11:20.565 --> 00:11:21.565
Back on track.

254
00:11:21.775 --> 00:11:22.815
Murphy's Law in Action.

255
00:11:23.995 --> 00:11:27.955
All right. Either one

256
00:11:27.955 --> 00:11:29.155
of you posting on teams for me,

257
00:11:29.815 --> 00:11:31.825
Working on it just had the startup teams.

258
00:11:35.765 --> 00:11:38.465
So we'll assume that the first question on the survey,

259
00:11:38.555 --> 00:11:41.505
Craig, isn't things went flawlessly during the webinar,

260
00:11:44.705 --> 00:11:45.325
If only,

261
00:11:50.705 --> 00:11:50.965
all right.

262
00:11:55.155 --> 00:11:58.255
That's funny. Yes, the AI has gone rogue.

263
00:11:58.675 --> 00:12:00.855
No, in this case it's just a teams feature.

264
00:12:01.485 --> 00:12:02.575
I've run into that before.

265
00:12:02.955 --> 00:12:04.535
Why won't it let me copy the link.

266
00:12:04.755 --> 00:12:06.855
It only lets me type it in or click it

267
00:12:11.135 --> 00:12:12.135
Link. I'm putting this

268
00:12:12.135 --> 00:12:13.345
link. Let me know if this one works.

269
00:12:14.355 --> 00:12:18.715
Don't, don't put that just yet 'cause I tried that one

270
00:12:18.855 --> 00:12:21.275
and it didn't work, so I'm gonna send you the correct link.

271
00:12:21.275 --> 00:12:23.675
Here we go. Alright, the last link I sent you.

272
00:12:23.675 --> 00:12:24.755
That's the one, Brodie.

273
00:12:24.905 --> 00:12:28.095
Perfect. Alright.

274
00:12:28.095 --> 00:12:29.335
Alright, so we're running this just

275
00:12:29.335 --> 00:12:31.055
so everybody knows on our staging environment,

276
00:12:31.055 --> 00:12:32.695
which you may notice in the link title.

277
00:12:33.115 --> 00:12:34.815
And the reason we're doing this is we're actually showing

278
00:12:34.815 --> 00:12:38.255
you something that we are in the process of releasing

279
00:12:38.275 --> 00:12:39.535
but just hasn't been through.

280
00:12:39.595 --> 00:12:41.255
Its, its final checks to make sure

281
00:12:41.255 --> 00:12:42.495
that everything's working perfectly.

282
00:12:42.955 --> 00:12:45.975
So this is a little bit of a sneak peek at software

283
00:12:45.975 --> 00:12:48.415
that is on the cusp of being released.

284
00:12:49.545 --> 00:12:52.355
Look, I put the, the updated link in there,

285
00:12:52.535 --> 00:12:54.035
so if everyone would try to click on it

286
00:12:54.055 --> 00:12:55.435
and let us, hopefully that's working.

287
00:12:56.455 --> 00:12:57.755
I'm trying to click on that now,

288
00:12:57.895 --> 00:13:00.755
hoping others have better luck than I did right now.

289
00:13:04.545 --> 00:13:06.035
Okay. Oh, that did it.

290
00:13:06.665 --> 00:13:08.795
Alright, we're back on track. Sorry about that.

291
00:13:09.285 --> 00:13:11.075
Sorry about that, that short interruption.

292
00:13:11.255 --> 00:13:13.355
But as I said, the last two questions are the,

293
00:13:13.355 --> 00:13:14.435
the most important ones there.

294
00:13:14.895 --> 00:13:17.315
Um, so I'll give everybody a moment to kind

295
00:13:17.315 --> 00:13:18.435
of think about an answer.

296
00:13:18.935 --> 00:13:20.635
And, and this is kind of important

297
00:13:20.635 --> 00:13:22.395
because the answers that you

298
00:13:23.275 --> 00:13:26.255
put into those last two questions are the kinds of things

299
00:13:26.255 --> 00:13:28.295
that we would expect to get from respondents.

300
00:13:28.295 --> 00:13:30.375
And if we have enough people who fill them in,

301
00:13:30.915 --> 00:13:33.855
it will hopefully, uh, make it much more impressive.

302
00:14:05.385 --> 00:14:07.195
Okay, this is good. I've already got 10

303
00:14:07.475 --> 00:14:08.915
responses they're starting to roll in.

304
00:14:08.995 --> 00:14:10.595
I know there's many more people than that.

305
00:14:30.505 --> 00:14:32.965
All right, so we have our survey delivered.

306
00:14:32.965 --> 00:14:33.965
What's next, Craig?

307
00:14:34.735 --> 00:14:38.385
Okay, so it looks like I have 16 responses

308
00:14:38.885 --> 00:14:39.945
and I think we have more than

309
00:14:39.945 --> 00:14:40.985
that number of people, don't we?

310
00:14:40.985 --> 00:14:43.305
Can you see how many people we have? Brodie like, uh, 21,

311
00:14:43.305 --> 00:14:45.185
there's still coming in pretty quickly. 33.

312
00:14:45.765 --> 00:14:47.705
Yep, there's, there's 33 people.

313
00:14:48.775 --> 00:14:50.665
Alright, I'll give it another

314
00:14:50.665 --> 00:14:51.875
moment just for people to type.

315
00:14:52.075 --> 00:14:53.275
'cause it does take a a minute

316
00:14:53.335 --> 00:14:55.035
to fill in a, a text response.

317
00:14:58.485 --> 00:15:01.695
Alright, so what I'm gonna do now is I'm going

318
00:15:01.695 --> 00:15:02.695
to share my screen

319
00:15:03.035 --> 00:15:04.375
and I'm gonna show you kind of

320
00:15:04.375 --> 00:15:06.255
what this would look out like on the backend.

321
00:15:07.035 --> 00:15:08.495
Um, so

322
00:15:17.665 --> 00:15:20.755
hopefully what you're seeing right now is our staging

323
00:15:20.755 --> 00:15:23.315
environment, and specifically you're looking at

324
00:15:23.715 --> 00:15:24.875
responses to a campaign.

325
00:15:25.055 --> 00:15:28.395
And so what this should look like on your screen is

326
00:15:28.395 --> 00:15:30.395
what somebody would see on the backend when they're looking

327
00:15:30.395 --> 00:15:31.555
at the responses from a survey.

328
00:15:32.175 --> 00:15:34.475
And so, you know, unsurprisingly, uh,

329
00:15:34.495 --> 00:15:38.635
we have a very high rate of response for, uh, question two,

330
00:15:38.855 --> 00:15:40.795
option D about today's session.

331
00:15:41.215 --> 00:15:43.315
Uh, just about everybody was at the session

332
00:15:43.935 --> 00:15:46.195
and, uh, you can see some people have also attended some

333
00:15:46.195 --> 00:15:47.275
others here and, and so forth.

334
00:15:47.455 --> 00:15:49.115
And this is the kind of more traditional stuff.

335
00:15:49.135 --> 00:15:50.715
So I'm just gonna scroll past it.

336
00:15:51.105 --> 00:15:53.195
Most people have seen this sort of thing before,

337
00:15:53.455 --> 00:15:56.355
but what's interesting, and I'm just going to very quickly

338
00:15:57.435 --> 00:15:58.995
reload in case there were some that came in late,

339
00:15:58.995 --> 00:16:01.315
which there were, I can see the answers

340
00:16:01.375 --> 00:16:02.995
to the two text questions down here,

341
00:16:03.455 --> 00:16:06.515
and I can say individual responses and I can show them.

342
00:16:06.855 --> 00:16:09.075
And you can see there's this massive responses

343
00:16:09.075 --> 00:16:10.755
and these are the things that you've all typed

344
00:16:11.015 --> 00:16:12.195
in response to this question.

345
00:16:12.615 --> 00:16:13.835
And I could look through this

346
00:16:13.855 --> 00:16:16.195
and I could see what everyone has said.

347
00:16:16.575 --> 00:16:19.955
Um, obviously we have a relatively small sample size here,

348
00:16:20.415 --> 00:16:22.115
and if I wanted to, I could probably

349
00:16:22.635 --> 00:16:24.995
manually without AI put this together in a few minutes.

350
00:16:25.495 --> 00:16:28.435
But, but you can get the idea that I can click, uh,

351
00:16:28.435 --> 00:16:29.635
SparkAI summary here.

352
00:16:29.775 --> 00:16:31.995
And, and this is the link that we've added the AI feature

353
00:16:32.375 --> 00:16:33.835
and it takes just a moment.

354
00:16:34.255 --> 00:16:35.915
And what it will do is it will go through

355
00:16:35.915 --> 00:16:39.555
and it's already got a, uh, kind of a, a summary here

356
00:16:39.575 --> 00:16:41.355
and you can see right here, uh,

357
00:16:41.385 --> 00:16:42.595
I'll zoom in a little bit just

358
00:16:42.595 --> 00:16:43.995
so the text is a little easier to read.

359
00:16:44.655 --> 00:16:46.275
And it's put together a summary

360
00:16:46.535 --> 00:16:48.235
of those individual responses.

361
00:16:52.845 --> 00:16:54.585
And I'm giving everybody a moment just to read

362
00:16:54.775 --> 00:16:56.265
what the AI has come up with.

363
00:17:00.625 --> 00:17:03.365
And obviously when we've got, you know, 20

364
00:17:03.425 --> 00:17:07.325
or 30 responses, it's, uh, it's mostly interesting

365
00:17:07.385 --> 00:17:08.565
as a, a proof of concept.

366
00:17:08.985 --> 00:17:12.085
But you could see how if you were in a situation where

367
00:17:12.625 --> 00:17:15.725
you had hundreds if not thousands of responses,

368
00:17:15.725 --> 00:17:18.645
compiling this information would be a real pain point.

369
00:17:19.045 --> 00:17:20.405
I mean, especially if this is a survey

370
00:17:20.405 --> 00:17:22.645
that maybe you give every semester or every year

371
00:17:22.705 --> 00:17:24.925
or every season with, with every test delivery,

372
00:17:25.235 --> 00:17:28.525
this gives you a way to very quickly get the, the gist of

373
00:17:28.595 --> 00:17:29.885
what your respondents are saying

374
00:17:29.885 --> 00:17:30.925
in response to the question.

375
00:17:31.185 --> 00:17:33.125
And just for completeness, I'll look at the second one.

376
00:17:33.125 --> 00:17:35.685
And you can see there, there are a bunch of responses here

377
00:17:35.705 --> 00:17:37.885
and you could scroll through and you could read them

378
00:17:38.025 --> 00:17:40.005
and you could look at them or you could just

379
00:17:40.005 --> 00:17:41.125
hit SparkAI summary.

380
00:17:41.915 --> 00:17:43.895
Um, now

381
00:17:45.545 --> 00:17:47.285
So, so Craig, you know,

382
00:17:48.555 --> 00:17:50.585
You said that this works really well when you

383
00:17:50.585 --> 00:17:52.065
have thousands of responses.

384
00:17:52.085 --> 00:17:54.625
We have 28, which isn't, which isn't bad.

385
00:17:55.125 --> 00:17:58.185
Can you tell us more about why the SparkAI summary

386
00:17:58.645 --> 00:17:59.705
is a link?

387
00:17:59.935 --> 00:18:02.345
I've been seeing others where there's a feature

388
00:18:02.345 --> 00:18:04.025
that I ask it questions instead.

389
00:18:04.285 --> 00:18:05.625
Uh, you know, let's share why,

390
00:18:06.125 --> 00:18:08.785
why SparkAI summary is, is just that?

391
00:18:09.585 --> 00:18:12.315
Sure. Uh, well we did consider alternatives like giving

392
00:18:12.335 --> 00:18:14.355
the user the ability to ask open-ended

393
00:18:14.555 --> 00:18:15.595
questions like you were saying.

394
00:18:15.895 --> 00:18:17.355
And we might do that down the line.

395
00:18:17.455 --> 00:18:19.275
Um, as the technology matures

396
00:18:19.495 --> 00:18:21.235
and AI gets better at understanding

397
00:18:21.255 --> 00:18:23.915
and responding to prompts, that will become a more

398
00:18:23.915 --> 00:18:25.075
and more attractive option.

399
00:18:25.975 --> 00:18:28.635
But for now, AI is still a little tricky to work with

400
00:18:28.935 --> 00:18:30.675
and there are some pitfalls that we wanted

401
00:18:30.675 --> 00:18:31.915
to help our clients avoid.

402
00:18:32.215 --> 00:18:33.995
Uh, by presenting users with a link,

403
00:18:34.175 --> 00:18:37.395
we maintain some control, control that helps us do that,

404
00:18:37.485 --> 00:18:40.355
helps our customers get the answers they want quickly,

405
00:18:40.355 --> 00:18:43.035
easily, and as consistently as possible.

406
00:18:44.185 --> 00:18:47.565
So, you know, let's, let's talk about, um,

407
00:18:47.755 --> 00:18:49.765
what those pitfalls are, uh,

408
00:18:49.765 --> 00:18:52.325
because I feel like that's really the strength in,

409
00:18:52.385 --> 00:18:55.125
in joining the session today is kind of learning from some

410
00:18:55.125 --> 00:18:57.765
of, some of our, uh, trials and errors.

411
00:18:57.985 --> 00:18:59.725
So, you know, what are those pitfalls?

412
00:19:00.525 --> 00:19:02.165
Absolutely, you're right. And, and I mean that's kind of

413
00:19:02.165 --> 00:19:04.565
where the thought process behind, you know,

414
00:19:04.565 --> 00:19:06.885
where we put AI into our software.

415
00:19:07.075 --> 00:19:08.845
What, what features do we pick and so forth.

416
00:19:09.185 --> 00:19:11.565
It has to bear in mind that there are some, some issues

417
00:19:11.565 --> 00:19:14.645
with the software, uh, with the ai, I mean with AI

418
00:19:14.645 --> 00:19:15.765
as a, as a concept.

419
00:19:16.305 --> 00:19:18.885
And you know, of course I'm talking about hallucinations,

420
00:19:18.885 --> 00:19:20.965
which just have to be part of the conversation

421
00:19:21.685 --> 00:19:23.925
whenever anyone talks about applications

422
00:19:24.225 --> 00:19:26.125
for the current crop of large language models.

423
00:19:26.505 --> 00:19:29.045
Uh, in short, sometimes the AI just goes off target

424
00:19:29.105 --> 00:19:30.285
and says things that aren't true.

425
00:19:30.905 --> 00:19:33.725
And in this context, if you can't trust the results,

426
00:19:34.235 --> 00:19:35.925
then they're not really much use to anyone.

427
00:19:36.665 --> 00:19:39.405
And to be fair, I would strongly recommend

428
00:19:39.405 --> 00:19:41.925
that anyone using anything generated

429
00:19:41.945 --> 00:19:44.685
by AI at least do a spot check of the results.

430
00:19:45.065 --> 00:19:47.605
Uh, the news is full of stories about people

431
00:19:47.605 --> 00:19:50.245
who got in trouble by trusting AI too much at this early

432
00:19:50.335 --> 00:19:52.405
stage in the technologies development.

433
00:19:52.585 --> 00:19:55.525
Uh, if you want a real disaster story about this, go ahead

434
00:19:55.525 --> 00:19:58.645
and Google Stephen Schwartz, he's an attorney who used AI

435
00:19:58.665 --> 00:20:00.045
to draft his legal brief

436
00:20:00.305 --> 00:20:01.325
and then he submitted it

437
00:20:01.325 --> 00:20:02.645
to the court without reading it first.

438
00:20:02.705 --> 00:20:04.845
And that went about as badly as you might think.

439
00:20:05.385 --> 00:20:06.685
Uh, but one of the things, and

440
00:20:06.685 --> 00:20:08.325
and the reason I've left this up on the screen

441
00:20:08.325 --> 00:20:10.725
for everybody here is I wanna point out

442
00:20:10.725 --> 00:20:13.005
that we still have the individual responses here,

443
00:20:13.425 --> 00:20:15.565
and that's there for a very important reason

444
00:20:15.565 --> 00:20:17.605
because you shouldn't completely trust the ai.

445
00:20:18.145 --> 00:20:22.685
It is absolutely fantastic for summarizing hundreds

446
00:20:22.685 --> 00:20:24.405
or even thousands of responses.

447
00:20:24.745 --> 00:20:26.445
But once you've got that AI summary,

448
00:20:26.825 --> 00:20:29.485
we still leave this here so that you can eyeball this data

449
00:20:29.705 --> 00:20:31.325
and, and that's not something that just applies

450
00:20:31.325 --> 00:20:33.725
to our software or to this component or anything.

451
00:20:34.605 --> 00:20:37.325
Anything you create with ai, you, you do have to check it.

452
00:20:37.625 --> 00:20:38.925
And I think that's important

453
00:20:39.105 --> 00:20:41.405
and it goes into our product process when we're thinking

454
00:20:41.405 --> 00:20:43.445
about, you know, how are we gonna present this information?

455
00:20:43.865 --> 00:20:46.125
It would be really, I think, negligent of us

456
00:20:46.125 --> 00:20:49.045
to give you the SparkAI summary without also giving you

457
00:20:49.145 --> 00:20:51.045
access to the individual responses

458
00:20:51.155 --> 00:20:53.485
that underlie those responses.

459
00:20:54.385 --> 00:20:57.165
So Craig, I had a question I two questions that came in

460
00:20:57.165 --> 00:20:59.845
through the, uh, q and a in the call.

461
00:20:59.845 --> 00:21:02.485
Sure. Uh, the first one is, is out of order.

462
00:21:02.505 --> 00:21:07.485
One is, um, hey, I'm using, uh, surveys with, uh,

463
00:21:07.565 --> 00:21:11.085
ITS right now and I don't see a SparkAI summary in mind.

464
00:21:11.505 --> 00:21:15.485
Um, you know, Craig, when, when is the SparkAI summary,

465
00:21:15.555 --> 00:21:18.245
when do we expect it to, to roll live? Uh,

466
00:21:18.245 --> 00:21:19.525
So this is in QA right now.

467
00:21:19.745 --> 00:21:21.405
Um, especially when it comes

468
00:21:21.405 --> 00:21:24.165
to implementing something involving ai,

469
00:21:24.185 --> 00:21:27.365
we wanna be responsible about making sure it's fully vetted

470
00:21:27.365 --> 00:21:29.205
and tested before we release it.

471
00:21:29.505 --> 00:21:31.605
So, um, this should be coming out, uh,

472
00:21:31.745 --> 00:21:33.085
let me see, it's August.

473
00:21:33.565 --> 00:21:35.485
I know that we do have a commitment

474
00:21:35.505 --> 00:21:37.445
to release this by October.

475
00:21:37.555 --> 00:21:38.805
Hopefully it will be sooner,

476
00:21:39.145 --> 00:21:40.605
but by then that that's a drop

477
00:21:40.605 --> 00:21:42.005
dead actually commitment that we have.

478
00:21:42.675 --> 00:21:45.125
Okay. And, and the second was the question about the

479
00:21:45.125 --> 00:21:49.605
surveys component, which is, um, will we be adding, uh,

480
00:21:49.925 --> 00:21:51.245
features to either do conditional

481
00:21:51.305 --> 00:21:53.125
or branching survey questions

482
00:21:53.515 --> 00:21:55.485
That is very, very close on the roadmap?

483
00:21:55.485 --> 00:21:57.845
Yes. That's actually something a little off topic for this,

484
00:21:58.145 --> 00:22:00.085
but uh, yeah, that's something I'm actually working on

485
00:22:00.085 --> 00:22:02.045
vision for right now. Um, alright,

486
00:22:02.195 --> 00:22:03.605
Well thank you for those questions.

487
00:22:04.225 --> 00:22:06.845
Um, audience, uh, we appreciate it.

488
00:22:07.675 --> 00:22:10.965
So let's, let's, um, you know, you are,

489
00:22:11.445 --> 00:22:14.805
I had interrupted you were talking about the, uh, kind

490
00:22:14.805 --> 00:22:18.365
of the, the tentative nature of knowing whether to trust,

491
00:22:18.785 --> 00:22:20.165
uh, SparkAI summaries.

492
00:22:20.705 --> 00:22:23.845
And I understand that nobody can eliminate those risk

493
00:22:23.845 --> 00:22:26.045
of the halluc hallucinations.

494
00:22:26.625 --> 00:22:30.005
Um, but what are we doing to minimize that

495
00:22:30.545 --> 00:22:32.165
and how are we making sure

496
00:22:32.165 --> 00:22:33.885
that those AI summaries are valid?

497
00:22:34.375 --> 00:22:36.365
Right. And and a lot of thought did go into that.

498
00:22:36.465 --> 00:22:39.685
So some of the risk we were able to avoid

499
00:22:40.185 --> 00:22:43.445
by our foundational decision about where to use ai, I mean,

500
00:22:43.445 --> 00:22:45.485
because there are a lot of ways people are using it these

501
00:22:45.485 --> 00:22:48.525
days, but when you ask a large language model a question

502
00:22:48.525 --> 00:22:50.845
where it needs to draw upon the entire universe

503
00:22:51.065 --> 00:22:53.925
of information upon which it was trained, there can be

504
00:22:53.985 --> 00:22:55.565
who knows what in there that nudges it

505
00:22:55.565 --> 00:22:56.645
toward a bizarre response.

506
00:22:56.645 --> 00:22:58.485
That's where you end up with hallucinations.

507
00:22:58.905 --> 00:23:02.245
But when you ask an LLM specifically to confine itself

508
00:23:02.265 --> 00:23:03.805
to a bounded set of data,

509
00:23:04.275 --> 00:23:06.485
that makes hallucinations much less likely.

510
00:23:07.065 --> 00:23:09.285
And, and so that's why, for example, uh, some

511
00:23:09.285 --> 00:23:11.085
of the mainstream uses you're seeing right now,

512
00:23:11.355 --> 00:23:13.365
like if you go to Amazon, they use AI

513
00:23:13.365 --> 00:23:15.325
to provide summaries of customer reviews.

514
00:23:15.665 --> 00:23:18.845
Why you can sometimes see meta AI summarizing long comment

515
00:23:18.875 --> 00:23:20.645
threads on Facebook.

516
00:23:20.985 --> 00:23:23.045
Uh, this is something that LLMs are

517
00:23:23.045 --> 00:23:24.645
inherently well-suited to do.

518
00:23:25.305 --> 00:23:27.045
And so by picking that, you know,

519
00:23:27.045 --> 00:23:29.085
we can't ever completely get rid of the risk,

520
00:23:29.505 --> 00:23:30.965
but we mitigate the risk

521
00:23:31.065 --> 00:23:33.605
by picking a task at which these particular

522
00:23:34.495 --> 00:23:36.135
programs are good.

523
00:23:36.675 --> 00:23:39.495
And also, once we picked the task that it was good at,

524
00:23:39.995 --> 00:23:41.575
we did some prompt engineering.

525
00:23:41.575 --> 00:23:42.935
Basically we sat down

526
00:23:43.035 --> 00:23:45.725
and we crafted a query that guides the AI

527
00:23:45.745 --> 00:23:48.205
to very specifically answer only the question

528
00:23:48.675 --> 00:23:50.365
that we want answered in the context

529
00:23:50.505 --> 00:23:52.765
of this survey question and these answers.

530
00:23:53.425 --> 00:23:55.045
And finally, our selection

531
00:23:55.045 --> 00:23:57.085
of language model also helps reduce the risk.

532
00:23:58.215 --> 00:24:01.195
All right, so, um, you know, you talked a,

533
00:24:01.245 --> 00:24:03.115
we've talked about it being a link.

534
00:24:03.655 --> 00:24:07.955
Uh, we've talked about, um, you know, hallucinations, um,

535
00:24:09.255 --> 00:24:11.995
people wanna know what query we're using, right?

536
00:24:12.065 --> 00:24:13.115
Sure. I mean, now we're getting

537
00:24:13.115 --> 00:24:14.195
into the nuts and bolts of it.

538
00:24:14.195 --> 00:24:15.395
Yeah. But I'm happy to share that.

539
00:24:15.735 --> 00:24:17.555
So it's, it's nothing too arcane, really.

540
00:24:17.575 --> 00:24:19.075
At least the end result looks simple.

541
00:24:19.175 --> 00:24:20.675
Uh, and I guess that's kind of some of the,

542
00:24:20.935 --> 00:24:23.995
the fun is making the, the difficult look simple, uh,

543
00:24:24.055 --> 00:24:26.395
the query is something like assume the survey asks the

544
00:24:26.515 --> 00:24:27.715
question and then we insert the stem

545
00:24:28.095 --> 00:24:29.635
and the responses are as follows.

546
00:24:29.635 --> 00:24:31.595
And then we list them and we say,

547
00:24:31.595 --> 00:24:33.835
without restating the original question first,

548
00:24:33.865 --> 00:24:35.475
provide a summary of these results.

549
00:24:35.775 --> 00:24:38.755
And then below that summary also provide a bulleted list

550
00:24:38.755 --> 00:24:41.715
of conclusions that can be validly drawn from these results.

551
00:24:42.375 --> 00:24:44.675
And that's it. And, and I think keeping it short actually

552
00:24:44.675 --> 00:24:46.315
helps in terms of of clarity.

553
00:24:47.215 --> 00:24:50.435
So, so we are trying to keep that prompt pretty much

554
00:24:50.455 --> 00:24:53.675
as short and specific as possible, right?

555
00:24:53.735 --> 00:24:57.875
Enough enough to keep it from repeating the question

556
00:24:58.255 --> 00:24:59.635
enough to give it some format.

557
00:24:59.895 --> 00:25:02.955
But we are not doing heavy prompt engineering on this one.

558
00:25:03.535 --> 00:25:06.315
So how do we arrive at that prompt?

559
00:25:06.685 --> 00:25:08.355
There actually was a fair amount of trial

560
00:25:08.355 --> 00:25:09.435
and error that went into it.

561
00:25:09.435 --> 00:25:11.155
So I don't know exactly what qualifies as

562
00:25:11.295 --> 00:25:12.715
as heavy prompt engineering.

563
00:25:12.935 --> 00:25:14.355
Um, but we knew we'd recognize a

564
00:25:14.355 --> 00:25:15.475
good response when we saw one.

565
00:25:15.735 --> 00:25:18.435
So it was a matter of trying different prompts over

566
00:25:18.435 --> 00:25:19.915
and over with different sets of data

567
00:25:20.215 --> 00:25:23.035
and also different LLMs until we found the combination

568
00:25:23.305 --> 00:25:25.595
that most consistently returned useful,

569
00:25:25.595 --> 00:25:26.835
correct, relevant answers.

570
00:25:27.335 --> 00:25:29.035
Uh, but someone who is familiar

571
00:25:29.035 --> 00:25:30.995
with LLMs might notice some traces

572
00:25:30.995 --> 00:25:32.995
of our process in the final prompt

573
00:25:32.995 --> 00:25:34.035
that I just described to you.

574
00:25:34.495 --> 00:25:36.715
Uh, the part where we ask the AI not

575
00:25:36.715 --> 00:25:39.835
to restate the original question, was the outcome of it kind

576
00:25:39.835 --> 00:25:41.235
of often choosing to do that.

577
00:25:41.575 --> 00:25:43.195
And since we knew based on

578
00:25:43.195 --> 00:25:44.835
where it was gonna be displayed in our software,

579
00:25:44.865 --> 00:25:46.075
that it would be right next

580
00:25:46.075 --> 00:25:47.475
to the question that wasn't useful.

581
00:25:47.935 --> 00:25:50.755
So we instructed the AI to omit it, the problem was solved,

582
00:25:50.775 --> 00:25:52.475
the answers got better, um,

583
00:25:52.495 --> 00:25:55.675
and it was just a bunch of small optimizations, refinements,

584
00:25:55.775 --> 00:25:57.875
ab testing to find the prompt

585
00:25:58.065 --> 00:26:00.475
that most consistently returned the best answer.

586
00:26:01.585 --> 00:26:03.565
All right, so Craig, we have more questions coming in

587
00:26:03.565 --> 00:26:06.085
and what what's interesting is that one of the questions

588
00:26:06.085 --> 00:26:09.645
that came in, um, was in line with my next question for you,

589
00:26:09.645 --> 00:26:11.165
which is that you had mentioned that choice

590
00:26:11.185 --> 00:26:14.565
of the large language model matters, which are we using?

591
00:26:14.705 --> 00:26:16.525
How do we get to that? Why are we using it?

592
00:26:17.115 --> 00:26:18.805
Sure. Um, so in answer

593
00:26:18.805 --> 00:26:23.525
to the question at the moment we're using chat GPT-3 0.5

594
00:26:23.855 --> 00:26:24.885
turbo instruct.

595
00:26:25.085 --> 00:26:26.205
I know that's quite a mouthful.

596
00:26:26.465 --> 00:26:29.165
Um, as you might gather from the name instruct

597
00:26:29.265 --> 00:26:32.045
or it's a model that's optimized specifically

598
00:26:32.065 --> 00:26:33.245
for staying on target

599
00:26:33.505 --> 00:26:36.485
and directly answering questions instead of getting kind

600
00:26:36.485 --> 00:26:38.725
of creative and conversational as many

601
00:26:38.725 --> 00:26:40.845
of the more general purpose LLMs tend to do.

602
00:26:41.185 --> 00:26:42.645
Um, it's all about getting to the point

603
00:26:42.665 --> 00:26:43.940
and providing the answer.

604
00:26:44.145 --> 00:26:46.845
And since that's exactly what we want for this application,

605
00:26:47.185 --> 00:26:48.325
that's the one we went with.

606
00:26:48.545 --> 00:26:50.245
But it's, it's important to notice also,

607
00:26:50.245 --> 00:26:53.245
or to note also that we can swap in a different model

608
00:26:53.265 --> 00:26:54.325
if a customer asks.

609
00:26:54.705 --> 00:26:56.925
And the technology honestly is moving so quickly

610
00:26:57.195 --> 00:26:59.045
that I'm sure there will be better models soon.

611
00:26:59.245 --> 00:27:01.605
I mean, and we definitely plan to take advantage of those

612
00:27:02.065 --> 00:27:04.805
as they become available, but for now, that was the one

613
00:27:04.805 --> 00:27:07.005
that was like the perfect tool for this job.

614
00:27:08.235 --> 00:27:10.735
And so, you know, putting something

615
00:27:10.735 --> 00:27:14.495
through a large language model, um, could cost something.

616
00:27:14.595 --> 00:27:17.415
So what are the costs? What do the costs look like, uh,

617
00:27:17.435 --> 00:27:20.135
to do this, uh, for these surveys, you know,

618
00:27:20.135 --> 00:27:21.255
for the survey component?

619
00:27:21.805 --> 00:27:23.135
Sure. Well, you're absolutely right.

620
00:27:23.275 --> 00:27:25.815
Um, they do have a cost per query and so forth.

621
00:27:26.035 --> 00:27:27.335
And, and right now, um,

622
00:27:27.455 --> 00:27:30.575
ITS is including the SparkAI feature at no additional cost.

623
00:27:30.955 --> 00:27:33.255
Um, it delivers a benefit to our customers.

624
00:27:33.555 --> 00:27:35.255
It addresses a common pain point

625
00:27:35.255 --> 00:27:36.775
with open-ended survey responses.

626
00:27:37.075 --> 00:27:39.815
And it allows our customers to get more familiar with

627
00:27:39.815 --> 00:27:43.415
how generative AI can help them in a relatively low stakes

628
00:27:43.415 --> 00:27:45.735
environment because yes, it does all the work,

629
00:27:46.235 --> 00:27:49.655
but they can then kind of scan those, those responses.

630
00:27:49.885 --> 00:27:51.495
It's not as high stakes as say,

631
00:27:51.495 --> 00:27:52.615
like generating items for a test.

632
00:27:52.615 --> 00:27:55.135
They can still scan through eyeball it,

633
00:27:55.135 --> 00:27:56.535
make sure it's approximately right.

634
00:27:57.155 --> 00:28:00.015
Uh, one of the things that we have done on the product side

635
00:28:00.015 --> 00:28:01.815
to control costs and at the same time

636
00:28:01.835 --> 00:28:04.775
to enhance performance is we've added a caching feature.

637
00:28:05.155 --> 00:28:08.775
So if a user requests an AI summary of a set of responses

638
00:28:08.885 --> 00:28:10.375
that has already been generated,

639
00:28:10.605 --> 00:28:12.415
then we return the stored answer instantly.

640
00:28:12.675 --> 00:28:15.095
Not only is that, uh, help in terms of response time,

641
00:28:15.715 --> 00:28:19.015
but uh, it it also means that we're not kind

642
00:28:19.015 --> 00:28:21.295
of running up the cost if someone clicks to the page

643
00:28:21.295 --> 00:28:22.895
and back and they're looking at it.

644
00:28:23.275 --> 00:28:26.375
But, um, anytime anyone clicks the link when there's even

645
00:28:26.435 --> 00:28:29.015
one new response, an updated answer is

646
00:28:29.015 --> 00:28:30.095
generated right on the spot.

647
00:28:32.435 --> 00:28:36.355
So, um, couple more questions have come in.

648
00:28:36.355 --> 00:28:38.875
We're gonna try to hit some of them here. Hey Jeffrey,

649
00:28:39.165 --> 00:28:41.875
Lemme, before you jump into the more questions,

650
00:28:41.975 --> 00:28:44.155
let me just, uh, separate two things

651
00:28:44.155 --> 00:28:45.595
that Craig just said since you're

652
00:28:45.595 --> 00:28:46.675
talking about cost on this.

653
00:28:47.145 --> 00:28:50.115
Sure. When Craig mentions that the feature will be,

654
00:28:50.125 --> 00:28:53.075
we're gonna enable it into the surveys component

655
00:28:53.175 --> 00:28:55.675
and it rolls out and if you're using it, you just,

656
00:28:55.675 --> 00:28:56.835
you'll get access to it.

657
00:28:57.425 --> 00:29:01.675
What we do when we roll out SparkAI for our customers is

658
00:29:01.705 --> 00:29:04.755
that you can test things in our environment,

659
00:29:04.815 --> 00:29:06.435
but you're gonna have to sign up

660
00:29:06.455 --> 00:29:09.475
for your own direct relationship with one of the LLMs

661
00:29:09.615 --> 00:29:12.235
and then provide those credentials to us for it.

662
00:29:12.705 --> 00:29:15.595
Your data, you need to think about your data,

663
00:29:15.725 --> 00:29:17.715
where does it live and, and so on.

664
00:29:17.815 --> 00:29:20.315
So those costs will actually be a direct thing and,

665
00:29:20.335 --> 00:29:22.915
and to be quite, and what Craig's mentioned is it's very

666
00:29:22.915 --> 00:29:26.315
minimal, but it is things that's gonna be on your side.

667
00:29:26.315 --> 00:29:28.555
But those are things we can talk about post webinar if you

668
00:29:28.555 --> 00:29:29.555
do have questions about that.

669
00:29:29.615 --> 00:29:32.995
But we're doing it for anything related to SparkAI

670
00:29:33.095 --> 00:29:35.835
for our integration from all the different pieces from

671
00:29:35.865 --> 00:29:38.475
content development, the surveys and so on.

672
00:29:39.015 --> 00:29:40.115
So thanks. Alright,

673
00:29:40.255 --> 00:29:41.255
Thanks.

674
00:29:41.305 --> 00:29:43.725
So, uh, Craig, there's a question that came in

675
00:29:43.725 --> 00:29:48.085
and said, are we going to use, uh, chat GPT-4, um,

676
00:29:49.315 --> 00:29:50.645
with the observation that

677
00:29:51.265 --> 00:29:54.205
it far exceeds three five in in many of the, you know,

678
00:29:54.205 --> 00:29:55.565
performance or quality metrics?

679
00:29:56.305 --> 00:29:57.925
So as I was saying

680
00:29:57.925 --> 00:30:02.005
before, we absolutely do intend as new models come out

681
00:30:02.425 --> 00:30:04.285
to use the new models, um,

682
00:30:05.065 --> 00:30:09.485
and we can configure the component for a specific client

683
00:30:09.905 --> 00:30:11.805
to use any model you want.

684
00:30:12.205 --> 00:30:14.725
I mean, once you set up an account as, as Brodie was saying,

685
00:30:14.875 --> 00:30:16.005
once you set up an account,

686
00:30:16.005 --> 00:30:18.005
we can make it powered by anything.

687
00:30:18.025 --> 00:30:20.605
So if you have a preference, we went with one

688
00:30:20.605 --> 00:30:24.165
that we thought was a good balance of, of cost and,

689
00:30:24.305 --> 00:30:26.125
and functionality for this purpose.

690
00:30:26.705 --> 00:30:29.405
But if you have a preference, we can absolutely do that.

691
00:30:29.785 --> 00:30:31.965
You know, and then there's another question that says,

692
00:30:32.385 --> 00:30:34.965
Hey, if we choose to use another L-L-L-L-M

693
00:30:35.185 --> 00:30:36.885
and pay for the API usage

694
00:30:36.885 --> 00:30:38.365
and provide the token, are there any

695
00:30:38.365 --> 00:30:39.525
additional costs to switching?

696
00:30:39.705 --> 00:30:41.365
And really the answer is no. Right?

697
00:30:41.705 --> 00:30:44.285
One of the, one of the things that we've built into all

698
00:30:44.285 --> 00:30:47.925
of our SparkAI integrations is the ability for a client

699
00:30:47.945 --> 00:30:50.565
to pick a model to us to be able to set it up.

700
00:30:50.745 --> 00:30:54.045
And then our software is using an internal API

701
00:30:54.045 --> 00:30:58.725
to then call these other APIs to then use those, uh,

702
00:30:58.905 --> 00:31:02.205
to use those instances or tenets of LLM.

703
00:31:02.505 --> 00:31:04.445
So I'm gonna mark one as answered live.

704
00:31:05.025 --> 00:31:06.645
Uh, I'm going to do another review.

705
00:31:07.105 --> 00:31:11.935
Um, let's see, uh, I have several questions here.

706
00:31:11.935 --> 00:31:14.135
We're gonna, we're gonna move on to the next one

707
00:31:14.335 --> 00:31:16.135
'cause it's the next question that I had for you

708
00:31:16.135 --> 00:31:17.455
and then we have time at the end.

709
00:31:17.875 --> 00:31:21.415
Uh, and that is, can you talk about the functionality,

710
00:31:21.655 --> 00:31:23.775
a AI functionality we're considering,

711
00:31:24.475 --> 00:31:27.215
and, um, adding to the, you know,

712
00:31:27.485 --> 00:31:30.455
what AI functionality are we considering adding

713
00:31:30.635 --> 00:31:31.855
to the survey component?

714
00:31:32.235 --> 00:31:35.855
Uh, because, you know, I have another couple questions, uh,

715
00:31:35.885 --> 00:31:39.935
with that regard of, uh, you know, are we considering, uh,

716
00:31:39.935 --> 00:31:43.375
doing analytics at SparkAI will return such as X amount

717
00:31:43.375 --> 00:31:46.735
of people had a sentiment of w while y amount

718
00:31:46.995 --> 00:31:49.455
of people had a sent, uh, sentiment Z.

719
00:31:50.585 --> 00:31:53.725
So it sounds to me from, from the way the question came in

720
00:31:53.725 --> 00:31:55.125
that people are interested in things that go

721
00:31:55.125 --> 00:31:56.325
beyond single questions.

722
00:31:56.585 --> 00:31:59.165
Yes. So absolutely that is something we're looking at.

723
00:31:59.305 --> 00:32:02.045
Uh, one of the things that, well, so we have tons of ideas

724
00:32:02.385 --> 00:32:04.085
and it's actually kind of exciting to be working

725
00:32:04.085 --> 00:32:06.645
with these powerful tools at a time when they're just kind

726
00:32:06.645 --> 00:32:08.365
of becoming practical to use

727
00:32:08.585 --> 00:32:10.685
and everybody's figuring out how to leverage them.

728
00:32:11.185 --> 00:32:13.525
Um, and so we started with what we have,

729
00:32:13.525 --> 00:32:14.565
what we've shown you today,

730
00:32:14.565 --> 00:32:16.365
because we saw it as low hanging fruit,

731
00:32:16.435 --> 00:32:18.765
it's a straightforward way it would work for everybody.

732
00:32:19.225 --> 00:32:21.045
It would ease one of the major pain points

733
00:32:21.045 --> 00:32:22.605
that customers have with surveys.

734
00:32:22.745 --> 00:32:24.165
So we were able to implement it.

735
00:32:24.345 --> 00:32:26.725
But as, as this rolls out, we are already looking ahead

736
00:32:26.725 --> 00:32:28.645
to some of the things that have come up in the chat.

737
00:32:28.985 --> 00:32:32.405
So, for example, the system was designed from the ground up

738
00:32:32.585 --> 00:32:35.725
to be kind of modular in terms of which AI we have

739
00:32:35.725 --> 00:32:37.485
behind the scenes doing this so

740
00:32:37.485 --> 00:32:38.925
that it wouldn't require an

741
00:32:38.965 --> 00:32:40.405
interface overhaul or anything like that.

742
00:32:40.465 --> 00:32:42.565
It would be just, oh, okay, we update the value for

743
00:32:42.565 --> 00:32:44.685
where it's pointing and now we're using a different

744
00:32:44.825 --> 00:32:46.085
AI to, to power it.

745
00:32:46.265 --> 00:32:47.645
So yes to that. Um,

746
00:32:47.705 --> 00:32:50.845
and we are actually looking at AI as an analysis

747
00:32:50.905 --> 00:32:53.205
for entire surveys, not just one question at a time.

748
00:32:53.785 --> 00:32:56.285
And, and we've already thought about how that might allow us

749
00:32:56.285 --> 00:32:58.045
to say, point out connections

750
00:32:58.045 --> 00:32:59.685
between answers across questions

751
00:32:59.795 --> 00:33:01.485
that might not be immediately obvious

752
00:33:01.585 --> 00:33:03.125
to a person looking at the same data.

753
00:33:03.465 --> 00:33:05.965
Uh, for example, an AI might point out a correlation

754
00:33:05.965 --> 00:33:07.365
between candidates ages

755
00:33:07.625 --> 00:33:09.765
and the tools they use to prepare for an exam,

756
00:33:10.025 --> 00:33:11.365
or maybe it'll highlight a connection

757
00:33:11.365 --> 00:33:12.765
between the data candidate testing

758
00:33:13.465 --> 00:33:16.125
and how they felt about conditions at the testing center,

759
00:33:16.255 --> 00:33:17.645
which might point toward an

760
00:33:17.645 --> 00:33:18.885
intermittent problem at the site.

761
00:33:19.205 --> 00:33:21.485
I mean, we can only guess what insights expand, you know,

762
00:33:21.485 --> 00:33:22.845
adding that functionality might yield

763
00:33:22.845 --> 00:33:25.365
because we don't know what questions it's gonna analyze yet.

764
00:33:25.985 --> 00:33:28.325
Um, another thing I think

765
00:33:28.325 --> 00:33:31.885
that we could add once the AI technology matures a bit is

766
00:33:31.905 --> 00:33:34.565
to go back to the original idea, uh,

767
00:33:34.645 --> 00:33:36.245
it'll become more feasible to have it

768
00:33:36.245 --> 00:33:37.525
where you can ask questions

769
00:33:37.945 --> 00:33:40.685
and say to an ai, Hey, based on the results of this survey,

770
00:33:41.515 --> 00:33:42.885
what about this, what about this?

771
00:33:42.985 --> 00:33:45.445
So ask questions either at the single question

772
00:33:45.865 --> 00:33:47.405
or at the full survey level.

773
00:33:47.905 --> 00:33:50.325
Um, but I think the reason we didn't do that in kind

774
00:33:50.325 --> 00:33:52.765
of the initial release right here is that

775
00:33:53.645 --> 00:33:56.525
I think AI will probably follow the trajectory of,

776
00:33:56.525 --> 00:33:58.565
of similar information access technologies.

777
00:33:58.875 --> 00:34:00.445
Like think back to search engines.

778
00:34:00.545 --> 00:34:03.365
I'm maybe, you know, if they're almost old now, uh,

779
00:34:03.385 --> 00:34:05.685
but I remember in the early days of search engines,

780
00:34:05.785 --> 00:34:06.965
if you misspelled something,

781
00:34:06.965 --> 00:34:08.325
you weren't gonna get useful results.

782
00:34:08.705 --> 00:34:11.965
And now I can destroy, I can destroy my query,

783
00:34:12.085 --> 00:34:13.565
I can mangle my typing in Google,

784
00:34:13.945 --> 00:34:16.165
and it generally figures out what I was really searching

785
00:34:16.165 --> 00:34:18.605
for, even if my spelling is atrocious or I typo.

786
00:34:19.105 --> 00:34:22.965
And I think AI models will almost certainly become better at

787
00:34:22.965 --> 00:34:24.685
responding intelligently to prompts

788
00:34:24.865 --> 00:34:27.885
and understanding the real thrust of the question.

789
00:34:28.815 --> 00:34:30.995
Um, and I think it'll happen in a very short time

790
00:34:30.995 --> 00:34:33.155
because there's certainly a lot of effort being kind

791
00:34:33.155 --> 00:34:34.195
of sent in that direction.

792
00:34:34.235 --> 00:34:35.395
A lot of money is going that way.

793
00:34:35.935 --> 00:34:39.115
Um, but I think that the technology is not quite there yet.

794
00:34:39.115 --> 00:34:41.835
Anybody who's played with an AI realizes that there is,

795
00:34:42.575 --> 00:34:44.835
you know, if, if you don't engineer your prompt carefully,

796
00:34:44.835 --> 00:34:46.195
you can get some pretty strange results.

797
00:34:46.815 --> 00:34:49.355
Uh, one of the other things that we've considered as an AI

798
00:34:49.355 --> 00:34:50.915
that could generate survey questions

799
00:34:50.915 --> 00:34:52.435
or even entire surveys for you,

800
00:34:53.095 --> 00:34:56.355
and when we consider AI applications, we kind of go

801
00:34:56.355 --> 00:34:58.515
through our traditional product process, which is

802
00:34:58.515 --> 00:35:00.875
to look at the opportunities and customer pain points

803
00:35:01.295 --> 00:35:03.755
and try and find places where we can improve workflows.

804
00:35:04.135 --> 00:35:06.795
Uh, part of the challenge with finding the right places

805
00:35:06.815 --> 00:35:09.115
to use AI is understanding the unique strengths

806
00:35:09.115 --> 00:35:10.355
and weaknesses of the technology,

807
00:35:10.815 --> 00:35:14.795
but hopefully, hopefully what we've done here is, is kind

808
00:35:14.795 --> 00:35:17.195
of given you some insight into the thought process.

809
00:35:17.375 --> 00:35:18.875
You know, why did we go with the link?

810
00:35:18.875 --> 00:35:19.915
Which engine did we use?

811
00:35:20.015 --> 00:35:21.515
How did we come up with the prompt so

812
00:35:21.515 --> 00:35:23.435
that you can see we're thinking about these things we're

813
00:35:23.435 --> 00:35:25.635
innovating and we're trying to keep ahead of the curve.

814
00:35:26.875 --> 00:35:30.225
So thank you so much, Craig, um,

815
00:35:30.485 --> 00:35:33.145
for answering my questions as awkwardly

816
00:35:33.165 --> 00:35:35.185
as they might come out of my lips.

817
00:35:35.445 --> 00:35:37.365
So, uh, we do have questions

818
00:35:37.365 --> 00:35:40.365
that are better worded from our participants though,

819
00:35:40.665 --> 00:35:43.365
so I'm gonna, I'm gonna try to ask some of those now.

820
00:35:43.365 --> 00:35:45.245
We've got time for, for several more.

821
00:35:45.305 --> 00:35:48.125
If, uh, if anyone wants to add them into the q

822
00:35:48.125 --> 00:35:51.805
and a, uh, first, um, from a legal perspective,

823
00:35:52.385 --> 00:35:55.165
do you have or will have a disclaimer

824
00:35:55.165 --> 00:35:59.125
that the received AI response should not be used verbatim?

825
00:35:59.545 --> 00:36:01.685
In other words, modifications should be made

826
00:36:01.825 --> 00:36:03.885
to the AI generated response?

827
00:36:06.385 --> 00:36:09.795
Well, um, so these responses, it's a good question really.

828
00:36:10.135 --> 00:36:13.995
Um, since the responses for surveys are generally intended

829
00:36:14.015 --> 00:36:18.185
to be used by the, by the customer, these are kind of things

830
00:36:18.185 --> 00:36:19.185
that are kind of backend.

831
00:36:19.605 --> 00:36:21.905
Um, I mean, I suppose somebody could publish them.

832
00:36:22.335 --> 00:36:23.545
It's, it's a good question, really.

833
00:36:23.725 --> 00:36:25.745
And, and I suppose, you know, this is one

834
00:36:25.745 --> 00:36:27.105
where I, I need to do some research.

835
00:36:27.175 --> 00:36:28.505
It's kind of funny, uh, most

836
00:36:28.505 --> 00:36:29.865
of our participants probably don't know.

837
00:36:29.885 --> 00:36:34.665
I'm also an attorney. Um, so I will probably have to go and,

838
00:36:34.765 --> 00:36:35.785
and look at some of that.

839
00:36:36.285 --> 00:36:39.425
But, you know, in terms of our role in product,

840
00:36:40.005 --> 00:36:41.945
it has been very much on getting the,

841
00:36:41.945 --> 00:36:43.145
the functionality there.

842
00:36:43.605 --> 00:36:45.785
Um, but yeah, probably some disclaimers

843
00:36:45.785 --> 00:36:46.985
are appropriate on Greg.

844
00:36:47.015 --> 00:36:48.665
It's a good question. And you know,

845
00:36:48.705 --> 00:36:50.905
I think about the SparkAI summary

846
00:36:50.975 --> 00:36:53.985
that it may be worth just when you click on it to just

847
00:36:55.145 --> 00:36:59.045
let people know that, uh, you know, it's AI generated

848
00:36:59.105 --> 00:37:02.995
and, you know, kind of be, you know,

849
00:37:03.665 --> 00:37:05.075
user beware type of thing.

850
00:37:05.175 --> 00:37:06.835
So, um, you know,

851
00:37:06.995 --> 00:37:09.595
I have a couple questions about the survey component in

852
00:37:09.595 --> 00:37:10.595
general that I'm gonna ask.

853
00:37:11.095 --> 00:37:14.835
One is, uh, does every user receive the same survey link

854
00:37:14.935 --> 00:37:17.075
or is it a unique link for every user?

855
00:37:17.865 --> 00:37:19.875
Okay, well that gets a little bit into the, um,

856
00:37:20.775 --> 00:37:23.115
That's, it depends, But I'm happy to answer it.

857
00:37:23.415 --> 00:37:27.275
So the way it works is if you want to send out a survey

858
00:37:27.295 --> 00:37:28.995
to a group of people and you're,

859
00:37:28.995 --> 00:37:30.035
they're basically anonymous,

860
00:37:30.265 --> 00:37:32.515
then you can use a link the way that we did today.

861
00:37:32.865 --> 00:37:36.875
However, you can also link a survey result

862
00:37:37.415 --> 00:37:39.755
to a test result or to a candidate.

863
00:37:40.095 --> 00:37:42.075
And the way you can do that is we have a system

864
00:37:42.305 --> 00:37:45.235
that wasn't really the focus of the presentation today, um,

865
00:37:45.255 --> 00:37:47.355
but I can actually, I can can actually show you that

866
00:37:47.415 --> 00:37:48.635
and I think that'll be more effective

867
00:37:48.635 --> 00:37:49.755
than just talking about it.

868
00:37:49.935 --> 00:37:51.395
So gimme a second, I'll share my screen

869
00:37:51.395 --> 00:37:52.395
and I'll show you how that works.

870
00:37:52.975 --> 00:37:54.665
Um, so

871
00:37:58.485 --> 00:38:00.755
going back into this page,

872
00:38:00.755 --> 00:38:04.275
which now you're looking at the survey responses, if I go

873
00:38:04.275 --> 00:38:05.555
to the campaign setting here,

874
00:38:05.715 --> 00:38:07.275
I can set up an automatic delivery rule.

875
00:38:07.735 --> 00:38:10.395
And so I can say, for example, a test

876
00:38:11.215 --> 00:38:14.515
and I can say I'm just picking a, a random test in a form,

877
00:38:14.735 --> 00:38:17.035
and I can either have it navigate to the survey at the end

878
00:38:17.035 --> 00:38:19.115
of the test or send an email sometime

879
00:38:19.125 --> 00:38:21.115
afterwards and I can save it.

880
00:38:21.415 --> 00:38:24.195
And when I do this, then I've now set up a rule

881
00:38:24.215 --> 00:38:25.515
and the survey will automatically

882
00:38:25.515 --> 00:38:26.555
be delivered with that test.

883
00:38:26.695 --> 00:38:29.275
And this is on staging, so it's not gonna affect any data,

884
00:38:29.305 --> 00:38:32.555
it's just a, a demo program and I can set up multiple rules

885
00:38:32.655 --> 00:38:34.115
and, and conditions and so forth.

886
00:38:34.615 --> 00:38:38.805
But when I do this, the test then will be delivered

887
00:38:38.905 --> 00:38:40.525
and it will be linked to that result.

888
00:38:40.945 --> 00:38:44.325
And so what you can then do is on the responses page here,

889
00:38:45.095 --> 00:38:46.435
you can download the raw data

890
00:38:46.535 --> 00:38:47.835
and when you download the raw data,

891
00:38:47.835 --> 00:38:50.315
you'll get an Excel spreadsheet that you can then see

892
00:38:50.655 --> 00:38:52.515
for a specific candidate ID

893
00:38:52.515 --> 00:38:54.035
and a specific test result,

894
00:38:54.185 --> 00:38:55.635
what did they answer to each question.

895
00:38:55.855 --> 00:38:56.955
And so then you can put

896
00:38:56.955 --> 00:38:59.035
that into any data analysis tool you want

897
00:38:59.295 --> 00:39:01.595
and you can get very granular information

898
00:39:01.595 --> 00:39:02.675
about those responses.

899
00:39:03.055 --> 00:39:06.755
So, so when we send it out in an email

900
00:39:06.855 --> 00:39:09.595
to the individual candidate, it is linked to the candidate

901
00:39:09.655 --> 00:39:13.155
and it is unique to that, to the, uh, recipient.

902
00:39:13.615 --> 00:39:15.995
Uh, so if we send it, if we're sending it out as email,

903
00:39:16.175 --> 00:39:17.315
it is linked to the recipient,

904
00:39:17.735 --> 00:39:22.075
but if we just use that URL, like the one that we use today,

905
00:39:22.575 --> 00:39:24.835
uh, and send out to everyone, those are all

906
00:39:25.465 --> 00:39:27.755
that is all anonymous, right? Exactly. That is anonymous

907
00:39:28.205 --> 00:39:29.995
Every time we create a campaign.

908
00:39:30.095 --> 00:39:32.155
And I guess now I'm getting into it.

909
00:39:32.175 --> 00:39:34.955
So the way that we have this set up is we actually have

910
00:39:35.505 --> 00:39:37.795
surveys and campaigns and they're separate concepts.

911
00:39:38.135 --> 00:39:41.195
Um, and so what we do is we have surveys.

912
00:39:41.415 --> 00:39:42.955
So you can see that, like for example,

913
00:39:42.955 --> 00:39:44.915
the survey here is the 8 26 demo survey,

914
00:39:45.135 --> 00:39:47.115
and that's the list of questions that everybody answered.

915
00:39:47.615 --> 00:39:49.515
And then we have a campaign, which is kind

916
00:39:49.515 --> 00:39:50.995
of a vehicle to launch the survey.

917
00:39:51.575 --> 00:39:55.995
And so if somebody wants to say, uh, send a group of surveys

918
00:39:56.135 --> 00:39:57.155
to different people,

919
00:39:57.585 --> 00:39:59.555
they can create multiple campaigns

920
00:39:59.615 --> 00:40:00.835
all based on the same survey.

921
00:40:01.175 --> 00:40:03.155
And that way they keep the responses kind

922
00:40:03.155 --> 00:40:04.515
of siloed in different groups.

923
00:40:04.815 --> 00:40:06.075
So you could send them to different classes

924
00:40:06.295 --> 00:40:08.555
or different years or however you wanted to do that.

925
00:40:09.295 --> 00:40:14.155
And so this link here is for everyone who will be taking it

926
00:40:14.155 --> 00:40:15.195
as part of this campaign.

927
00:40:15.415 --> 00:40:17.635
And this rule also will return people,

928
00:40:17.725 --> 00:40:20.595
their responses will be counted as part of this campaign,

929
00:40:20.595 --> 00:40:22.715
even if I had others set up with different rules.

930
00:40:23.415 --> 00:40:27.645
Um, so that's all kind of, that lets you kind of, uh,

931
00:40:27.645 --> 00:40:29.765
group your responses by, you know,

932
00:40:29.765 --> 00:40:31.205
which group of people received it.

933
00:40:31.425 --> 00:40:32.565
But the, the short answer

934
00:40:32.565 --> 00:40:35.445
to your question is you can either send this link,

935
00:40:35.445 --> 00:40:37.765
which is anonymous and every campaign has one,

936
00:40:38.225 --> 00:40:40.445
or you can set up rules that will send them,

937
00:40:40.465 --> 00:40:41.925
and then we'll include a link back

938
00:40:41.925 --> 00:40:43.085
to the person who responded.

939
00:40:44.925 --> 00:40:49.655
Okay. So, uh, more questions, um, maybe I missed it,

940
00:40:50.035 --> 00:40:53.695
but will the cache be handled on I S'S side or on our,

941
00:40:54.395 --> 00:40:56.095
or our own org?

942
00:40:56.525 --> 00:40:57.775
It's handled by ITS.

943
00:40:57.775 --> 00:40:59.615
Is there a limit on the cache such

944
00:40:59.615 --> 00:41:01.375
that it can exceed x gigabytes

945
00:41:01.675 --> 00:41:03.175
and will be stored for y months?

946
00:41:03.405 --> 00:41:04.895
Also, will this, uh,

947
00:41:05.105 --> 00:41:07.415
cache be available throughout multiple sessions

948
00:41:07.415 --> 00:41:09.335
with different LLM models?

949
00:41:09.875 --> 00:41:12.095
So I'm gonna, I'm gonna try to answer this one Craig,

950
00:41:12.115 --> 00:41:13.375
and then you can correct me.

951
00:41:13.755 --> 00:41:15.895
Uh, but you were mentioning that when I run

952
00:41:15.895 --> 00:41:19.095
that SparkAI summary, it's going out making the call

953
00:41:19.435 --> 00:41:21.615
to the LLM, bringing back the response,

954
00:41:21.755 --> 00:41:23.495
and then we are caching it locally.

955
00:41:24.485 --> 00:41:26.855
That is only expired out

956
00:41:26.955 --> 00:41:29.855
by there being additional responses to the survey.

957
00:41:30.515 --> 00:41:32.295
Is that correct? Yeah, and it's only,

958
00:41:32.555 --> 00:41:35.445
and it's only caching that the SparkAI summary.

959
00:41:35.745 --> 00:41:39.085
So we're really only talking about, you know, very small,

960
00:41:39.185 --> 00:41:40.685
you know, kilobits worth of data.

961
00:41:42.135 --> 00:41:44.105
Yeah, exactly. So there really is no limit

962
00:41:44.105 --> 00:41:46.345
because you're not gonna have a lot of text.

963
00:41:46.365 --> 00:41:47.505
The whole point of the summary is

964
00:41:47.505 --> 00:41:49.145
that it should be a relatively brief response.

965
00:41:49.485 --> 00:41:52.345
Um, it is done to the question, uh,

966
00:41:52.645 --> 00:41:53.825
to the point asked in the question.

967
00:41:54.045 --> 00:41:56.425
It is done on I s's side, we do that for you.

968
00:41:56.645 --> 00:41:57.825
So that means

969
00:41:57.825 --> 00:42:01.905
that we're storing in our system the AI summary, uh,

970
00:42:02.085 --> 00:42:03.665
and in fact it's done per program.

971
00:42:03.845 --> 00:42:06.545
So if there were two or three people at your organization

972
00:42:06.565 --> 00:42:07.905
who were all clicking around

973
00:42:07.925 --> 00:42:09.425
and looking at the results of a survey

974
00:42:09.485 --> 00:42:12.585
and so forth, you wouldn't be seeing the cost go up

975
00:42:13.045 --> 00:42:15.505
as people were clicking it, you know, on different screens.

976
00:42:15.855 --> 00:42:19.025
Once the AI summary has been generated, it's cached

977
00:42:19.205 --> 00:42:20.625
and it's basically bookmarked.

978
00:42:20.645 --> 00:42:25.065
As long as there aren't any additional responses to

979
00:42:25.065 --> 00:42:29.485
that question, you will see the same summary each time.

980
00:42:29.985 --> 00:42:32.085
As soon as there's at least one more answer to

981
00:42:32.085 --> 00:42:35.245
that question, then SparkAI summary will have

982
00:42:35.245 --> 00:42:36.485
to take into account the new question.

983
00:42:36.705 --> 00:42:38.765
So it reposes that question to the ai.

984
00:42:39.225 --> 00:42:40.925
All right, and I have, it looks like we

985
00:42:40.925 --> 00:42:42.005
have a couple minutes left.

986
00:42:42.205 --> 00:42:46.965
I have one question that's in, uh, in the q and a left, um,

987
00:42:47.665 --> 00:42:49.045
and I will read it for us.

988
00:42:49.345 --> 00:42:51.085
Can you share what new capabilities,

989
00:42:51.525 --> 00:42:53.005
features are on the dev roadmap?

990
00:42:53.105 --> 00:42:56.285
For example, administering survey questions before or

991
00:42:56.285 --> 00:43:00.205
after a segment directly in the site, not a survey link,

992
00:43:00.275 --> 00:43:02.885
different question types, matrix ratings,

993
00:43:03.275 --> 00:43:07.845
different response option types, images, basis, et cetera,

994
00:43:08.765 --> 00:43:11.885
recommending survey questions perhaps based on SparkAI

995
00:43:12.155 --> 00:43:15.765
summary, next Gen 2.0, question mark. Um,

996
00:43:16.345 --> 00:43:18.565
So some of those, Let's take a one at a time.

997
00:43:19.845 --> 00:43:23.385
Um, but I can tell you that we are going

998
00:43:23.405 --> 00:43:26.385
to integrate in very, very short order,

999
00:43:26.495 --> 00:43:27.545
like we're working on it

1000
00:43:27.545 --> 00:43:30.065
to come out at the same time as this AI tool.

1001
00:43:30.065 --> 00:43:33.425
So in the next month or so, the ability to deliver, uh,

1002
00:43:33.445 --> 00:43:34.865
say startup segment type things.

1003
00:43:34.865 --> 00:43:37.145
And so that's coming out very, very soon.

1004
00:43:37.205 --> 00:43:39.305
That's, I I believe we have client commitments.

1005
00:43:39.305 --> 00:43:42.465
So that's happening. Um, you know, with respect

1006
00:43:42.465 --> 00:43:45.505
to matrix questions that is already in development.

1007
00:43:45.645 --> 00:43:46.985
So that's, you know, that's,

1008
00:43:46.985 --> 00:43:48.145
that's no longer on the roadmap.

1009
00:43:48.145 --> 00:43:50.905
That's in process. Um, that is the next release

1010
00:43:50.905 --> 00:43:52.985
that's coming after what you see on the screen.

1011
00:43:53.605 --> 00:43:57.145
Um, the release after that is at this point,

1012
00:43:57.425 --> 00:43:58.905
although now we're talking two releases out.

1013
00:43:59.165 --> 00:44:01.545
So we reserve the right to adjust a little bit, uh,

1014
00:44:01.545 --> 00:44:03.585
in response to demand from our customers.

1015
00:44:03.965 --> 00:44:06.425
But that one will include branching, uh, the idea

1016
00:44:06.425 --> 00:44:07.425
that you can set up, you know,

1017
00:44:07.425 --> 00:44:08.785
different flows in your survey

1018
00:44:08.885 --> 00:44:10.345
so you can skip questions based on

1019
00:44:10.345 --> 00:44:11.745
earlier responses and things like that.

1020
00:44:12.245 --> 00:44:14.985
I'm sorry, Jeffrey, that was a long list and I didn't Yeah,

1021
00:44:14.985 --> 00:44:15.985
It is. And it, I

1022
00:44:15.985 --> 00:44:17.785
feel like Brodie's gonna cut us off, so, yeah,

1023
00:44:17.785 --> 00:44:18.785
I got it. We gotta end this now.

1024
00:44:18.785 --> 00:44:19.905
So thank you, um,

1025
00:44:20.365 --> 00:44:22.505
for the person asking the question, just check

1026
00:44:22.505 --> 00:44:23.785
with your program management team

1027
00:44:23.785 --> 00:44:26.505
and then we can share their, uh, their current, uh,

1028
00:44:26.505 --> 00:44:29.545
current customer and we're, we're happy to talk to you that,

1029
00:44:29.635 --> 00:44:30.905
about that stuff directly.

1030
00:44:31.275 --> 00:44:33.525
Well, thank you all so much for joining.

1031
00:44:34.265 --> 00:44:36.405
We, we really appreciate it.

1032
00:44:36.465 --> 00:44:39.285
We hope you enjoy just learning more about the surveys

1033
00:44:39.305 --> 00:44:41.325
and hopefully you can see the next,

1034
00:44:41.475 --> 00:44:42.845
next screen that I have coming up.

1035
00:44:42.845 --> 00:44:44.125
So thank you for joining this session.

1036
00:44:44.345 --> 00:44:45.885
We, we, we hope it gave you a lot

1037
00:44:45.885 --> 00:44:49.205
to think about some new ideas, new approaches to surveys,

1038
00:44:49.465 --> 00:44:50.645
and we're really excited about

1039
00:44:50.645 --> 00:44:51.845
what's happening on the roadmap.

1040
00:44:51.905 --> 00:44:54.565
So it'll be as you are diving into it.

1041
00:44:54.945 --> 00:44:57.765
So please join us next week for our final session.

1042
00:44:57.995 --> 00:44:59.125
It's gonna be on Thursday,

1043
00:44:59.125 --> 00:45:02.685
August 15th at 1:00 PM Eastern Daylight savings time.

1044
00:45:03.065 --> 00:45:05.125
So please join and listen to Chris Glacken

1045
00:45:05.125 --> 00:45:08.805
our Director of Innovative Technology, as he discussed

1046
00:45:08.985 --> 00:45:11.725
how to mitigate AI security threats.

1047
00:45:12.025 --> 00:45:14.085
Ugh, it's gonna be a juicy one.

1048
00:45:14.825 --> 00:45:17.405
So it's gonna be a mid-level a, uh, session.

1049
00:45:17.945 --> 00:45:20.125
And then when this, when this ends,

1050
00:45:20.125 --> 00:45:21.925
please don't click out too quick.

1051
00:45:22.025 --> 00:45:25.205
We really want you to take the survey on here

1052
00:45:25.305 --> 00:45:28.405
so we can get feedback and, and keep bringing you better

1053
00:45:28.405 --> 00:45:30.005
and better, uh, webinar content.

1054
00:45:30.575 --> 00:45:32.445
Thank you so much, and thank you Craig,

1055
00:45:32.465 --> 00:45:34.845
and thank you, Jeffrey, for hosting today's session.

1056
00:45:35.735 --> 00:45:37.405
Thank you everyone. Have a great afternoon.