← All Webinars | L.A.B.S. #4

1
00:00:09.380 --> 00:00:12.650
Hello everybody, and thank you so much for joining us today.

2
00:00:13.310 --> 00:00:16.530
We are gonna give everyone a few minutes to join. Before we get started,

3
00:00:16.840 --> 00:00:21.530
I'll aim to get started around 12:03 because we have quite the jam packed

4
00:00:21.530 --> 00:00:25.970
session in store. But, uh, while we're waiting for everyone to join,

5
00:00:26.510 --> 00:00:30.890
we have got a poll here for you to answer. So I'm gonna launch that now.

6
00:00:33.240 --> 00:00:37.820
Uh, the panelists and I are a little curious to know a little more about how you

7
00:00:37.850 --> 00:00:39.620
feel about remote proctoring today.

8
00:00:40.400 --> 00:00:43.660
So please go ahead and share your thoughts as you're getting settled in,

9
00:00:43.960 --> 00:00:47.500
and I will share the results with you before we dive in.

10
00:01:04.480 --> 00:01:07.590
Great, lots of you answering the poll. More than half. Appreciate that.

11
00:01:32.680 --> 00:01:36.290
Once again, if you are just joining us, we're gonna get started at around 12:03,

12
00:01:36.350 --> 00:01:39.770
but in the meantime, uh, please get settled in and, uh,

13
00:01:40.560 --> 00:01:41.810
take a look at the poll we've launched.

14
00:01:41.820 --> 00:01:46.090
We're trying to get a sense of how you feel about re remote proctoring today.

15
00:01:59.610 --> 00:02:04.240
75% of you hit have answered so far, which I, which I really appreciate.

16
00:02:20.660 --> 00:02:23.120
You'll notice that there's not really a neutral option on here,

17
00:02:23.140 --> 00:02:25.360
so we are really trying to force you to make a decision.

18
00:02:52.810 --> 00:02:55.700
Alright, we're at 12:03, so I'm going to stop the poll.

19
00:02:55.700 --> 00:02:58.960
82% of you have participated and,

20
00:02:59.000 --> 00:03:01.820
and I don't know if it's a surprise or not, but, uh,

21
00:03:02.990 --> 00:03:07.400
most of you have a pretty positive feel toward remote proctoring overall.

22
00:03:07.400 --> 00:03:11.640
Either you love it or the benefits outweigh the challenges.

23
00:03:11.780 --> 00:03:14.520
So I think that's really helpful for us to know,

24
00:03:14.950 --> 00:03:18.120
kind of migrating into, into the webinar today.

25
00:03:19.730 --> 00:03:24.030
So thank you so much. Okay, well,

26
00:03:24.080 --> 00:03:28.070
we're gonna get started. So, welcome everyone. My name is Sandy Hogg.

27
00:03:28.170 --> 00:03:30.990
I'm the Manager of Business Development at ITS,

28
00:03:31.570 --> 00:03:35.230
and I will be your host and moderator during today's webinar. Once again,

29
00:03:35.230 --> 00:03:38.990
thank you so much for joining us. Uh, we really do appreciate your,

30
00:03:38.990 --> 00:03:39.950
your time and your company.

31
00:03:41.310 --> 00:03:45.700
We've got a great session lined up today with an impressive panel where we'll be

32
00:03:45.700 --> 00:03:50.380
trying to touch on all areas of security considerations for remote

33
00:03:50.380 --> 00:03:53.660
proctored exams. So we're hoping that you leave with, you know,

34
00:03:53.660 --> 00:03:57.860
some new ideas and a rounded perspective on securing these exams

35
00:03:58.480 --> 00:04:00.620
before, during, and after test day.

36
00:04:01.270 --> 00:04:04.330
And that's also how we try to group our questions as well to,

37
00:04:04.350 --> 00:04:07.130
to make sure that we're addressing this topic holistically.

38
00:04:08.400 --> 00:04:10.090
This will be a panel style webinar,

39
00:04:10.230 --> 00:04:13.130
so for the majority of the time it'll just be me and the panelists on screen

40
00:04:13.130 --> 00:04:16.560
talking. Before we jump in,

41
00:04:16.760 --> 00:04:20.120
I just wanted to go over a few housekeeping items. First.

42
00:04:20.180 --> 00:04:24.280
we have reserved some dedicated time at the end of this webinar for Q and A,

43
00:04:24.740 --> 00:04:25.720
but you know, that said,

44
00:04:25.730 --> 00:04:29.600
we're not going to be just ignoring any questions that are submitted during the

45
00:04:29.600 --> 00:04:32.240
webinar as a rule or anything. So, you know,

46
00:04:32.240 --> 00:04:36.840
if something that one of our guests says does inspire a question that makes more

47
00:04:36.840 --> 00:04:39.880
sense in context, please do go ahead and submit it using that,

48
00:04:39.880 --> 00:04:41.200
that Q and A button in your,

49
00:04:41.200 --> 00:04:43.960
your Zoom interface and we'll try to work it in as time allows.

50
00:04:45.220 --> 00:04:46.590
This webinar will be recorded,

51
00:04:46.810 --> 00:04:49.870
so we'll share the link to access that after the webinar has ended.

52
00:04:49.870 --> 00:04:54.670
You'll get an email and there will also be a survey at the end, uh,

53
00:04:54.670 --> 00:04:58.110
that will appear in your Zoom window, so please don't click away too quickly.

54
00:04:58.530 --> 00:04:58.860
Uh,

55
00:04:58.860 --> 00:05:02.550
we'd love to hear your feedback about your thoughts on today's webinar as well

56
00:05:02.550 --> 00:05:05.760
as suggestions for future topics. Alright,

57
00:05:06.140 --> 00:05:07.680
so without any further delay,

58
00:05:07.880 --> 00:05:12.560
I am going to ask today's panelists to join and introduce themselves.

59
00:05:18.620 --> 00:05:21.800
All right. Everyone's very prompt. Uh,

60
00:05:22.420 --> 00:05:27.040
so I'm going to start with Pete. Um, I'll ask you to introduce yourself,

61
00:05:27.040 --> 00:05:28.440
tell us a little bit about yourself,

62
00:05:28.900 --> 00:05:32.520
and I'm wondering if you'll also share a bit about what you're currently focused

63
00:05:32.520 --> 00:05:37.520
on and sort of what's top of mind for you from a security standpoint in just a

64
00:05:37.520 --> 00:05:38.353
few words.

65
00:05:38.810 --> 00:05:40.880
Thank you, Sandy. Uh, my name is Pete Van Dyke.

66
00:05:40.880 --> 00:05:44.800
I've been working in certification for 15 years. Uh,

67
00:05:45.020 --> 00:05:49.480
I'm a security program manager at Amazon with their certification program,

68
00:05:50.100 --> 00:05:52.240
uh, spent the last three and a half years doing that,

69
00:05:52.340 --> 00:05:57.280
and I divide my time currently between people that steal our content and publish

70
00:05:57.280 --> 00:06:00.280
it on brain dump websites, uh, proxy testers,

71
00:06:00.460 --> 00:06:03.880
and those that misbehave on individual exams.

72
00:06:05.310 --> 00:06:09.260
Great, thank you so much. Um, Harry, why don't you go next?

73
00:06:09.960 --> 00:06:11.420
Hi, I am Harry Samit. Uh,

74
00:06:11.460 --> 00:06:15.020
I am the director of Special Investigations for Pearson VUE. Uh,

75
00:06:15.050 --> 00:06:20.020
I've been in certifications and assessment for about three years. Prior to that,

76
00:06:20.220 --> 00:06:23.380
I spent 21 years, uh, as a Special Agent in the FBI,

77
00:06:23.380 --> 00:06:27.340
and then before that nine years as a Naval Intelligence Officer. Um,

78
00:06:27.640 --> 00:06:31.180
my typical day-to-day is using the techniques I learned in those,

79
00:06:31.390 --> 00:06:34.940
those past professions, um, to,

80
00:06:34.980 --> 00:06:38.500
to figure out what happened when there's a, an incident, a crisis misconduct,

81
00:06:39.280 --> 00:06:41.180
um, of some nature on an exam,

82
00:06:41.180 --> 00:06:45.710
either discovered in advance during the exam or after the exam. Um,

83
00:06:45.770 --> 00:06:47.190
the team that I work with, uh,

84
00:06:47.190 --> 00:06:50.630
we try and figure out what happened and who is responsible. Great.

85
00:06:51.630 --> 00:06:54.620
Thank you. Jon, how about you?

86
00:06:56.400 --> 00:06:58.780
Thanks, Sandy. My name is Jon Jensen.

87
00:06:59.000 --> 00:07:02.180
I'm the Senior Vice President of Product Management at Examity.

88
00:07:02.730 --> 00:07:06.820
I've been in the industry for just a couple of years, so a relative newcomer,

89
00:07:06.820 --> 00:07:10.020
but I've been in the software and technology space for over 20 years.

90
00:07:10.080 --> 00:07:11.500
So really fun.

91
00:07:12.080 --> 00:07:15.780
My role is really thinking about how to make our software work better,

92
00:07:16.320 --> 00:07:20.140
and that's a challenge because I'm trying to balance the needs of our test

93
00:07:20.160 --> 00:07:24.020
takers and making sure that they have a smooth and seamless experience while

94
00:07:24.130 --> 00:07:28.460
also making sure that we are enforcing the rules of, um,

95
00:07:28.460 --> 00:07:30.660
the program and making sure that the exam is,

96
00:07:30.800 --> 00:07:33.620
is a fair experience for everybody. So thanks for having me.

97
00:07:34.490 --> 00:07:38.400
Great. Thanks for joining us. And last but not least, Chris.

98
00:07:39.690 --> 00:07:41.990
Hi, thanks, Sandy. My name's Chris Glacken.

99
00:07:41.990 --> 00:07:44.510
I'm Director of Innovative Technologies at ITS.

100
00:07:44.980 --> 00:07:47.390
I've been in the assessment industry for about nine years now,

101
00:07:47.850 --> 00:07:51.830
and my main focus is on, um, building out our, uh, secure browsers,

102
00:07:51.830 --> 00:07:54.510
our lockdown browsers for the Windows and Mac platforms,

103
00:07:54.970 --> 00:07:59.030
as well as building out our ITS ProctorNow, um, remote proctoring product.

104
00:08:00.330 --> 00:08:04.600
Great. Awesome. We have, you guys have such a, a, a,

105
00:08:04.720 --> 00:08:08.160
a wide range of experience, so I'm really excited to dive in here. Uh,

106
00:08:08.220 --> 00:08:12.040
I'd like to start with, uh, sort of an introductory high level question.

107
00:08:12.140 --> 00:08:16.480
So we're talking about, you know, defending ourselves against, um, you know,

108
00:08:16.480 --> 00:08:20.000
de de defending ourselves, uh, in a remote proctoring scenario.

109
00:08:20.220 --> 00:08:24.560
So I'm curious about, you know, the enemies that we're defending against. Right?

110
00:08:24.580 --> 00:08:29.000
So what motivations could a bad actor have, uh,

111
00:08:29.040 --> 00:08:32.800
during a remote remotely proctored exam? Uh, I'd like to start with Harry.

112
00:08:34.570 --> 00:08:38.590
So in terms of motivation, uh, I think the obvious one, um,

113
00:08:38.730 --> 00:08:42.430
and the one that we see most often, probably the vast majority, uh,

114
00:08:42.430 --> 00:08:46.750
of bad actors is gonna be financial. Um, like any criminal, um,

115
00:08:47.410 --> 00:08:49.990
the folks who seek to hack,

116
00:08:50.620 --> 00:08:54.430
hack or steal intellectual property, um, you know,

117
00:08:54.450 --> 00:08:57.640
to do all the sorts of things they do to impact an exam program,

118
00:08:57.870 --> 00:09:00.520
they're doing it to make money. Um, you know,

119
00:09:00.550 --> 00:09:04.720
it's a pretty lucrative business out there, uh, depending on the desirability,

120
00:09:05.420 --> 00:09:07.880
uh, and the potential for changing, uh,

121
00:09:07.900 --> 00:09:12.640
the path of someone's life of a certification. Um, you know, the,

122
00:09:12.660 --> 00:09:16.520
the bad guys stand to make a substantial profit from, um,

123
00:09:16.750 --> 00:09:21.200
influence and exam and exam the course of an exam. There are two other,

124
00:09:21.660 --> 00:09:23.960
um, motivations we've detected and,

125
00:09:24.020 --> 00:09:27.800
and they're rare enough that it's a surprise when they happen. Um,

126
00:09:28.020 --> 00:09:32.000
one of them is, uh, is altruism. One of them is they will, uh,

127
00:09:32.200 --> 00:09:35.600
people will often steal intellectual property because they think they're

128
00:09:35.600 --> 00:09:40.040
helping, um, other candidates, uh, you know, they don't intend to sell it,

129
00:09:40.040 --> 00:09:43.600
they don't intend to profit financially f from it. Um, but they think they're,

130
00:09:43.600 --> 00:09:47.960
they're helping their colleagues or their, their co travelers in that space.

131
00:09:48.740 --> 00:09:53.680
Um, and then the other kind, uh, that is probably the rarest, fortunately, um,

132
00:09:53.740 --> 00:09:58.500
is educational misconduct where an instructor will actually, you know,

133
00:09:58.500 --> 00:10:03.500
someone who is responsible for doing training will actually steal content, uh,

134
00:10:03.500 --> 00:10:06.860
because it's easier to do that rather than actually teaching his students, uh,

135
00:10:06.920 --> 00:10:09.220
you know, the materials that they need to pass the exam.

136
00:10:10.480 --> 00:10:14.860
Oh, interesting. All right. So money altruism and educational misconduct.

137
00:10:15.280 --> 00:10:16.980
Uh, I'm gonna go over to Pete and see,

138
00:10:16.980 --> 00:10:19.180
do you have anything to add to that list or expand upon?

139
00:10:19.810 --> 00:10:21.460
Well, uh, on the financial side,

140
00:10:21.610 --> 00:10:24.860
just in the IT exam universe,

141
00:10:25.410 --> 00:10:30.140
it's estimated that proxy testing is a multi hundred million

142
00:10:30.160 --> 00:10:34.060
dollar business annually. So, uh, you know, the, uh,

143
00:10:34.520 --> 00:10:38.060
the organizations that are doing this are very sophisticated, uh,

144
00:10:38.290 --> 00:10:42.020
very motivated, and it's a very lucrative business for them. So, uh,

145
00:10:42.020 --> 00:10:43.420
this isn't going away anytime soon.

146
00:10:44.970 --> 00:10:48.510
And could you define proxy testing just for the audience in case there's someone

147
00:10:48.510 --> 00:10:49.630
who doesn't know what that means here?

148
00:10:50.100 --> 00:10:54.150
Sure. Proxy testing is when a, uh, individual or an organization, uh,

149
00:10:54.210 --> 00:10:58.870
offers to take an exam for a candidate for a fee. Uh,

150
00:10:59.250 --> 00:11:02.030
and we see a wide variety of fees for this. On the IT side,

151
00:11:02.030 --> 00:11:03.350
it's not unusual for a,

152
00:11:03.810 --> 00:11:08.670
for fees of between $250 and $2,000 for

153
00:11:08.710 --> 00:11:10.750
a, uh, for a proxy exam. Wow. So, uh,

154
00:11:10.810 --> 00:11:14.070
if somebody doesn't feel confident enough or if they've got some other

155
00:11:14.070 --> 00:11:14.830
motivation,

156
00:11:14.830 --> 00:11:19.030
I need to have this certification in order to keep my job as an example, uh,

157
00:11:19.700 --> 00:11:20.550
it's, uh, there,

158
00:11:20.550 --> 00:11:22.750
there are plenty of people out there that are willing to take a test,

159
00:11:22.750 --> 00:11:25.510
usually through remote control of a computer. Uh,

160
00:11:25.730 --> 00:11:27.150
and that's known as proxy testing.

161
00:11:28.010 --> 00:11:31.830
Wow. Thank you. Appreciate that. Alright,

162
00:11:31.830 --> 00:11:33.430
so now that we've set the stage a little bit,

163
00:11:33.850 --> 00:11:36.630
I'd like to move into thinking about, you know,

164
00:11:36.630 --> 00:11:40.390
some things that need to happen prior to a candidate sitting down to take a

165
00:11:40.390 --> 00:11:43.870
test, maybe during sort of the design phase of an exam or a form.

166
00:11:45.030 --> 00:11:47.220
First question here is, you know,

167
00:11:47.220 --> 00:11:52.020
what should programs be thinking about as they are designing and planning for,

168
00:11:52.240 --> 00:11:56.840
uh, an exam that they'd like to be remote proctored? You know, for example,

169
00:11:56.840 --> 00:12:00.640
what are some industry best practices for securing those remotely proctored

170
00:12:00.730 --> 00:12:04.480
exams? I'm gonna start with Chris, I'm curious for your perspective on this.

171
00:12:07.250 --> 00:12:10.140
Sure. So I think with regards to that question,

172
00:12:10.140 --> 00:12:13.900
three things really jump out at me When I think about that. The,

173
00:12:13.920 --> 00:12:16.740
the things that you wanna be thinking about are, uh, a level,

174
00:12:16.770 --> 00:12:18.900
what level of security you're gonna want to implement,

175
00:12:19.490 --> 00:12:21.620
what do you want the examinee experience to be,

176
00:12:22.040 --> 00:12:26.220
and what kind of resources do you have to implement everything? So,

177
00:12:26.320 --> 00:12:29.540
so I think all three of those put together really going to, um,

178
00:12:29.920 --> 00:12:30.980
put you in a spot where,

179
00:12:31.190 --> 00:12:33.820
where you end up with whatever solution that you're gonna go for.

180
00:12:33.920 --> 00:12:36.340
So during that process, you're gonna want to ask your thing,

181
00:12:36.340 --> 00:12:37.980
you're going to ask yourself things like,

182
00:12:38.200 --> 00:12:42.060
how well do you trust your candidates and what are the penalties for cheating a

183
00:12:42.180 --> 00:12:45.340
a penalty? Like, um, being, being banned from the possession, uh,

184
00:12:45.340 --> 00:12:48.700
being banned from a profession for life is gonna be a lot more severe than a

185
00:12:48.700 --> 00:12:52.700
more lower stakes college course that isn't gonna have something like that and

186
00:12:52.700 --> 00:12:55.620
they can go turn around and take the test or move into another course.

187
00:12:55.760 --> 00:12:59.340
So mm-hmm. So those are factors that come into play. And, um,

188
00:12:59.440 --> 00:13:02.580
you need to ask yourself things like, what, what is your plan if you're,

189
00:13:02.640 --> 00:13:05.940
if you get content exposure and are, are you even okay with the,

190
00:13:06.040 --> 00:13:10.140
the possibility of having content exposure delivering through remote?

191
00:13:10.540 --> 00:13:12.460
'cause that is a possibility. Um,

192
00:13:12.480 --> 00:13:16.060
you need to think through the processes that you're gonna have for violations

193
00:13:16.120 --> 00:13:20.480
and potential misconduct and, um, and,

194
00:13:20.900 --> 00:13:23.600
and also the, uh, the possibility of misconduct.

195
00:13:23.660 --> 00:13:25.960
So what are you gonna do if it's not a hundred percent certain?

196
00:13:25.960 --> 00:13:28.120
Because most of the time it's probably not gonna be,

197
00:13:28.180 --> 00:13:30.440
or be able to a hundred percent prove it without a doubt.

198
00:13:30.820 --> 00:13:34.360
So those are things that you're all gonna want to think of. Um, with,

199
00:13:34.360 --> 00:13:36.720
with regards to best practices, you're gonna want to, you,

200
00:13:36.780 --> 00:13:40.000
you wanna look at things like definitely a lockdown browser is a must have.

201
00:13:40.020 --> 00:13:41.560
It should not be the only thing you have,

202
00:13:41.580 --> 00:13:45.880
but it should be one of those levels that you have with your security to camera

203
00:13:46.190 --> 00:13:49.520
room scans, um, verifying the candidate and those type of things as well.

204
00:13:51.990 --> 00:13:54.500
Thank you, Chris. Um, Harry,

205
00:13:54.500 --> 00:13:57.500
curious if you have anything to add to this in terms of best practices?

206
00:13:58.340 --> 00:14:03.220
I do. So we, we think of, we think of exam security for, uh,

207
00:14:03.220 --> 00:14:06.660
frankly for in-person or remotely proctored exams as a,

208
00:14:06.660 --> 00:14:11.100
as a 360 degree endeavor. Um, by which we mean before, during,

209
00:14:11.320 --> 00:14:15.350
and after the exam. Um, you know, best practices,

210
00:14:15.970 --> 00:14:20.750
as Chris said, really depend on the appetite for risk, um, the threat that the,

211
00:14:20.750 --> 00:14:25.750
that the exam sponsor is facing. Um, but, but prior to the exam, um,

212
00:14:25.860 --> 00:14:28.870
that incorporates things like, uh, exam design, you know,

213
00:14:28.890 --> 00:14:32.910
how the content is presented and security features inherent in the design of

214
00:14:32.910 --> 00:14:36.270
the, of the test itself during the exam. Um,

215
00:14:36.370 --> 00:14:40.430
it incorporates things like Chris said, like the, uh, the, the secure browser,

216
00:14:40.850 --> 00:14:45.310
um, greeter and proctor training, um, steps that, uh,

217
00:14:45.450 --> 00:14:47.470
the exam provider and, you know,

218
00:14:47.470 --> 00:14:51.680
the exam delivery mechanism use to keep the exam secure

219
00:14:52.410 --> 00:14:56.320
while the candidate is in the hot seat. And then the, the final phase,

220
00:14:56.420 --> 00:14:59.400
the after is really where my team comes in, uh, most often,

221
00:14:59.720 --> 00:15:04.320
although we certainly dabble in, in all three arenas. Um, and that is, um,

222
00:15:04.830 --> 00:15:09.720
when misconduct or the possibility of misconduct is detected, uh,

223
00:15:09.980 --> 00:15:13.520
we step in and then conduct a thorough investigation to find out what happened.

224
00:15:13.700 --> 00:15:15.320
And to Chris's point, um,

225
00:15:15.320 --> 00:15:19.560
there are very few black and whites in the exam security world. Uh,

226
00:15:19.750 --> 00:15:24.120
it's not generally simple to be able to say, okay,

227
00:15:24.120 --> 00:15:28.320
we understand with complete clarity what happened, and we can give you,

228
00:15:28.940 --> 00:15:32.720
uh, a blow by blow. Um, there are some ambiguities here. Uh,

229
00:15:32.740 --> 00:15:36.920
and that's where experience and corporate knowledge and understanding the

230
00:15:36.920 --> 00:15:38.880
threat, what the bad guys do and how they operate,

231
00:15:38.880 --> 00:15:40.920
that's where that all comes together. Hmm.

232
00:15:42.240 --> 00:15:45.580
So 360 degree endeavor I like, I, I like,

233
00:15:45.780 --> 00:15:50.200
I think that's helpful to think of it that way. Alright.

234
00:15:50.340 --> 00:15:53.480
So n next I'd like to, to,

235
00:15:54.020 --> 00:15:58.160
I'm curious about how much information you share with the end users, you know,

236
00:15:58.160 --> 00:16:01.120
the candidates to prepare for testing. You know,

237
00:16:01.260 --> 00:16:05.720
how do you set expectations with them and is there a risk of giving too much

238
00:16:06.040 --> 00:16:09.840
information away in that sort of candidate preparation, um,

239
00:16:09.840 --> 00:16:14.760
that might aid bad actors or those who who wish to misbehave?

240
00:16:14.980 --> 00:16:16.160
Um, I'm gonna start with Pete.

241
00:16:17.210 --> 00:16:18.640
Thank you. Uh, well, we,

242
00:16:18.740 --> 00:16:22.760
we at Amazon try to be as transparent as possible as far as what kind of

243
00:16:23.000 --> 00:16:27.760
behaviors will result in an invalidation, uh,

244
00:16:28.060 --> 00:16:31.640
of an exam or perhaps banning from a program. So, uh,

245
00:16:31.740 --> 00:16:36.000
we do this in multiple ways and multiple spaces, including a, uh,

246
00:16:36.000 --> 00:16:37.640
checklist that has to be, uh,

247
00:16:37.800 --> 00:16:41.760
attested to right before launch of a proctored exam. So, uh,

248
00:16:41.760 --> 00:16:44.640
we tell 'em that you can't leave the, uh, the webcam view,

249
00:16:44.740 --> 00:16:47.440
you can't lean partially off screen, you can't have a third party,

250
00:16:47.540 --> 00:16:50.360
you can't access an electronic device mm-hmm. Or cell phone,

251
00:16:50.370 --> 00:16:55.200
those types of things. Uh, what we've found is, uh, is that you, you can't be,

252
00:16:55.700 --> 00:16:59.920
you can't, you can't deliver this information frequently enough. Uh, as,

253
00:16:59.940 --> 00:17:03.040
as you've, as you probably know, if you've ever taken one of these exams,

254
00:17:03.140 --> 00:17:06.200
you're nervous, you're skipping over things that say, click here to continue.

255
00:17:06.720 --> 00:17:09.000
Uh, so even if we put it 10 times, they use,

256
00:17:09.100 --> 00:17:12.720
the chances of people skipping over 10 times are pretty frequent. Now, what we,

257
00:17:12.720 --> 00:17:15.560
uh, what what we do slightly differently, we, uh,

258
00:17:15.580 --> 00:17:20.200
we inform our candidates that we use data forensics after the fact to look at

259
00:17:20.480 --> 00:17:24.600
score similarities and other, uh, other categories. Uh,

260
00:17:24.820 --> 00:17:29.240
we are not entirely transparent on what it is that we look for with data

261
00:17:29.480 --> 00:17:33.440
forensics, because that would allow people, uh, an edge and give them a way to,

262
00:17:33.580 --> 00:17:37.000
uh, to misbehave without detection. So, uh,

263
00:17:37.170 --> 00:17:41.160
while we do talk about the fact that we're using these tools and what the

264
00:17:41.360 --> 00:17:42.640
sanctions are for them, we don't, uh,

265
00:17:42.660 --> 00:17:45.640
we don't specify what exactly we do with data forensics.

266
00:17:45.980 --> 00:17:50.960
Mm-hmm. Yeah. You don't wanna give give all that away. Um, all right,

267
00:17:50.960 --> 00:17:53.280
great. I'm curious, Jon, do you have anything to add?

268
00:17:55.250 --> 00:17:55.680
Yeah,

269
00:17:55.680 --> 00:17:58.820
one of the things that's always a challenge for our program is to make sure that

270
00:17:58.820 --> 00:18:02.300
you're clearly communicating the rules and expectations during the exam.

271
00:18:02.760 --> 00:18:04.740
And just like Pete said, people don't like to read.

272
00:18:05.280 --> 00:18:06.860
So one of the biggest challenges,

273
00:18:07.280 --> 00:18:12.020
how to distill what really matters into a short and consumable set

274
00:18:12.200 --> 00:18:15.300
of rules to help test takers really understand.

275
00:18:15.960 --> 00:18:20.180
So we definitely continue to repeat and refresh, um, test takers,

276
00:18:20.180 --> 00:18:22.100
keep 'em in every notification that we can,

277
00:18:22.100 --> 00:18:24.860
helping them understand that there are rules, you should pay attention to them.

278
00:18:25.400 --> 00:18:29.460
But from the program's perspective, make sure that, like Chris said earlier,

279
00:18:29.490 --> 00:18:32.540
that you've thought about kind of end-to-end what rules are really impactful,

280
00:18:33.320 --> 00:18:37.860
and make sure that you're enforcing those with your, um, proctoring provider,

281
00:18:38.400 --> 00:18:41.780
but also make sure that you're not making it overly complicated. Mm-hmm. Um,

282
00:18:41.780 --> 00:18:44.420
you have a lot of flexibility with different platforms than what you can do,

283
00:18:44.560 --> 00:18:47.020
but make sure you have a plan ahead of time for what you should do,

284
00:18:47.020 --> 00:18:51.700
what really matters. Things like, um, if a child walks into the room or not,

285
00:18:51.700 --> 00:18:53.220
if a pet walks into the room or not,

286
00:18:53.220 --> 00:18:55.140
like some of those things can be really disruptive,

287
00:18:55.490 --> 00:18:58.180
potentially in a remote testing environment.

288
00:18:58.180 --> 00:19:00.700
We all don't have the same control of our environment as you do in the test

289
00:19:00.700 --> 00:19:01.320
center.

290
00:19:01.320 --> 00:19:04.700
So you've gotta think about what really is going to matter to your program and

291
00:19:04.700 --> 00:19:07.460
what's not. Mm-hmm. And that comes down to your assessment of the risk,

292
00:19:07.800 --> 00:19:08.820
how it impacts the program,

293
00:19:09.040 --> 00:19:11.620
and how it impacts the experience of the person taking the test.

294
00:19:11.880 --> 00:19:16.070
Mm-hmm. Great. Thanks. Harry, you have your hand up, you have something to add?

295
00:19:16.260 --> 00:19:16.550
Yeah,

296
00:19:16.550 --> 00:19:19.910
I just wanted to add it, it's an interesting question how much I,

297
00:19:20.970 --> 00:19:24.750
how much information is shared with test takers? With candidates? Um,

298
00:19:24.810 --> 00:19:27.350
that's a good one because it's always a balance, right? When,

299
00:19:27.380 --> 00:19:31.430
when taking action to remediate misconduct exam,

300
00:19:31.430 --> 00:19:33.990
sponsors wanna be able to tell the candidate, you know,

301
00:19:33.990 --> 00:19:35.790
we got you dead to rights, and here's why.

302
00:19:36.020 --> 00:19:38.070
Obviously the biggest motivation for that is to,

303
00:19:38.250 --> 00:19:41.990
for stall any legal action or any, you know,

304
00:19:42.130 --> 00:19:46.590
the candidate taking any steps to, you know, defame, publicly defame the,

305
00:19:46.610 --> 00:19:51.350
the exam sponsor. But the reality is, as that information is shared, um,

306
00:19:51.350 --> 00:19:54.390
these techniques spread through the illicit testing community.

307
00:19:54.390 --> 00:19:57.470
And so the bad guys learn more and more about how we do business. And so,

308
00:19:59.290 --> 00:20:02.390
um, a, a very famous icon in the testing industry,

309
00:20:02.410 --> 00:20:05.950
one that I was presenting with at a conference once said, um, in,

310
00:20:05.970 --> 00:20:09.630
in language I understood candidates are not entitled to due process. Right?

311
00:20:09.660 --> 00:20:11.870
This isn't a criminal proceeding.

312
00:20:11.870 --> 00:20:15.430
And so we don't have to tell them every piece of evidence that we used in how we

313
00:20:15.710 --> 00:20:17.270
obtained it before we take action. Mm-hmm.

314
00:20:17.450 --> 00:20:20.470
And so that's a great thing to remember, right? We, you know,

315
00:20:20.490 --> 00:20:22.630
as a security practitioner, uh,

316
00:20:22.790 --> 00:20:27.550
I wanna share much less typically than than usual because I don't want them to

317
00:20:27.550 --> 00:20:28.383
know how we do business.

318
00:20:30.120 --> 00:20:32.270
Right? So then that's an interesting question is like,

319
00:20:32.270 --> 00:20:36.750
how much evidence is enough to share with them that like, we, we've got you,

320
00:20:36.750 --> 00:20:39.910
we know it, but without kind of giving too much away.

321
00:20:40.450 --> 00:20:42.310
And of course, the candidate knows, right? I mean,

322
00:20:42.310 --> 00:20:45.750
they have guilty knowledge so they know, right? So, you know, for all of the,

323
00:20:46.250 --> 00:20:48.560
um, posturing and the lawsuits that,

324
00:20:48.560 --> 00:20:51.000
that I've seen threaten in the three years that I'm here,

325
00:20:51.270 --> 00:20:54.120
that I've been here in, in this, in this industry, um,

326
00:20:54.280 --> 00:20:58.740
I have yet to see a candidate actually sue, uh, you know, an exam provider,

327
00:20:58.750 --> 00:21:01.580
especially when the evidence, when they know the evidence is ample,

328
00:21:01.580 --> 00:21:03.860
even if it hasn't been explained to them. Hmm.

329
00:21:05.030 --> 00:21:09.450
That's definitely good to keep in mind. Awesome. Well, thank you so much.

330
00:21:09.500 --> 00:21:12.530
We're gonna shift a little bit into, you know,

331
00:21:13.120 --> 00:21:15.530
when the candidate's in the hot seat to use Harry's words,

332
00:21:15.600 --> 00:21:17.930
sort of during the testing, um,

333
00:21:17.930 --> 00:21:21.530
some security tactics or some things to be thinking about. Um,

334
00:21:21.550 --> 00:21:22.730
so one important thing that we,

335
00:21:22.790 --> 00:21:26.970
we talk about here is balancing user experience with security, right?

336
00:21:26.970 --> 00:21:30.090
This is something that I think everyone who's delivering a remotely proctored

337
00:21:30.090 --> 00:21:32.330
exam is trying to do. Um,

338
00:21:32.830 --> 00:21:37.650
so what are some factors to consider when setting that balance?

339
00:21:38.150 --> 00:21:39.290
Uh, I'd like to start with Jon.

340
00:21:41.300 --> 00:21:44.270
Yeah, thank you. There's a lot that goes into that.

341
00:21:44.370 --> 00:21:48.470
And when you think about the design of a platform or a tool set,

342
00:21:49.050 --> 00:21:51.630
the hard work is in making a complicated process,

343
00:21:51.820 --> 00:21:56.110
like checking in and taking an exam be simple and easy for the test taker.

344
00:21:56.490 --> 00:21:59.830
So when we think about the user experience, there's a couple of key motivations.

345
00:22:00.490 --> 00:22:01.170
One is,

346
00:22:01.170 --> 00:22:05.550
how do we get the test taker into the hot seat as quickly as possible?

347
00:22:06.370 --> 00:22:09.750
How do we make sure that they understand exactly what they're going to encounter

348
00:22:09.760 --> 00:22:13.750
along the way? So, for example, if there's an ID check,

349
00:22:13.750 --> 00:22:16.870
they're expected to present a government issue to ID along the way,

350
00:22:17.210 --> 00:22:19.310
we need to make sure that they know that and they expect it.

351
00:22:19.310 --> 00:22:22.790
Because you'd be surprised how many times people are like, oh, sorry,

352
00:22:22.790 --> 00:22:26.830
I gotta go get my wallet. Oh, my wallet's in my car. Oh, my, um,

353
00:22:26.830 --> 00:22:29.510
partner took my car to the grocery store. You know, I mean, there's,

354
00:22:29.580 --> 00:22:31.590
it's amazing how many times that happens in the moment.

355
00:22:32.090 --> 00:22:36.670
And the longer that process takes, the more stress there is for the test taker.

356
00:22:36.730 --> 00:22:39.750
Mm-hmm. Um, the other consideration is the technical problems.

357
00:22:39.750 --> 00:22:41.910
If your technology doesn't work smoothly,

358
00:22:42.540 --> 00:22:45.990
there's gonna be points in time where there's inevitably some sort of a

359
00:22:45.990 --> 00:22:49.150
technical problem, which really can amplify the stress on the candidate,

360
00:22:49.320 --> 00:22:51.390
which means ultimately, even if they get into the test,

361
00:22:51.740 --> 00:22:55.190
they might not have the same ability to perform as they did before.

362
00:22:55.330 --> 00:22:59.430
So we're thinking about how do we make it predictable? How do we make it smooth?

363
00:22:59.690 --> 00:23:03.510
And on the other side of that, from an internal perspective,

364
00:23:03.520 --> 00:23:04.670
we're always looking at the metrics.

365
00:23:05.130 --> 00:23:08.030
What's the time it takes test takers to check in? Where do they get stuck?

366
00:23:08.030 --> 00:23:12.270
Where do our support tickets get filed so that internally we can find and assess

367
00:23:12.270 --> 00:23:15.390
those and make changes to our platform to remove those, um,

368
00:23:15.410 --> 00:23:16.590
or reduce those barriers.

369
00:23:17.520 --> 00:23:21.050
Mm-hmm. Yeah, you wanna be as frictionless as possible,

370
00:23:21.230 --> 00:23:25.370
but with also making sure everything's set up. Um, Chris,

371
00:23:25.510 --> 00:23:27.930
I'm curious if you have anything to add there about, you know,

372
00:23:27.930 --> 00:23:31.490
how to set that balance between user experience and, and security.

373
00:23:32.570 --> 00:23:36.940
Sure. Yeah, I, I agree with everything Jon said. Um, to kind of expand on,

374
00:23:37.000 --> 00:23:41.420
on a, a bit from a technical perspective, uh, an example is, um, I,

375
00:23:41.540 --> 00:23:43.820
I like what Jon said about making sure a candidate is,

376
00:23:43.840 --> 00:23:45.800
is prepared and they know what's gonna be expected of them.

377
00:23:46.220 --> 00:23:49.680
So a security feature that I've really come to find value in is a,

378
00:23:49.680 --> 00:23:53.800
is a two camera implementation. So you want to, when you do security,

379
00:23:53.820 --> 00:23:56.720
you always wanna make sure you have certain levels. So if they get by one level,

380
00:23:56.980 --> 00:23:59.440
you have other things in place that are gonna be there to try to catch that.

381
00:23:59.440 --> 00:24:02.720
And two cameras a good way to work around those kind of things.

382
00:24:02.820 --> 00:24:05.160
But also with regards to user experience,

383
00:24:05.600 --> 00:24:08.480
introducing a second camera also introduces another friction point.

384
00:24:08.710 --> 00:24:11.920
It's a piece of technology and connection, so it's another point of failure too.

385
00:24:12.340 --> 00:24:14.400
So you need to think, so the candidate needs to be prepared,

386
00:24:14.400 --> 00:24:15.440
know that they're gonna have that,

387
00:24:15.630 --> 00:24:18.120
know that they're gonna have a device for it and know that they're gonna need to

388
00:24:18.120 --> 00:24:21.120
have a spot and how they're gonna need to position that because it's gonna be a

389
00:24:21.120 --> 00:24:23.920
smaller device, need a way to put it. Um,

390
00:24:23.940 --> 00:24:27.840
accommodations are also something that I find very interesting from a security

391
00:24:27.840 --> 00:24:31.160
perspective. So accessibility software, um,

392
00:24:31.680 --> 00:24:33.400
anything an accessibility software is gonna do.

393
00:24:33.460 --> 00:24:37.720
All those inter interfaces it can have are something, another, uh, process,

394
00:24:38.120 --> 00:24:40.560
a malicious process that somebody can make, could use too.

395
00:24:40.900 --> 00:24:43.760
So you don't want to keep the door open completely to accessibility,

396
00:24:43.820 --> 00:24:47.040
but you wanna make sure that you're offering and making available the

397
00:24:47.040 --> 00:24:49.200
accessibility tools to the people that actually need it.

398
00:24:49.460 --> 00:24:51.840
So you wanna take those things into consideration as well.

399
00:24:52.660 --> 00:24:54.360
So you're saying that if, you know,

400
00:24:54.800 --> 00:24:59.080
allowing other programs or other processes to be running for accessibility

401
00:24:59.080 --> 00:24:59.790
purposes,

402
00:24:59.790 --> 00:25:04.120
invites sort of opens more doors for something to be happening with those,

403
00:25:04.310 --> 00:25:05.600
like someone to take over it?

404
00:25:06.180 --> 00:25:07.400
So processes, yes.

405
00:25:07.460 --> 00:25:11.840
But also it's more the fact that those processes are using native APIs with the

406
00:25:11.840 --> 00:25:13.920
system. So if you're gonna allow those APIs in general,

407
00:25:13.920 --> 00:25:17.280
whether it's those processes or a process that's homegrown,

408
00:25:17.410 --> 00:25:20.480
those are the things that you need to consider should you really be allowing

409
00:25:20.480 --> 00:25:22.440
those APIs to interact with your application.

410
00:25:23.480 --> 00:25:26.890
Okay. Because it could pose greater risks. Okay.

411
00:25:26.990 --> 00:25:30.050
It can, yeah, it can be used for, um, content manipulation,

412
00:25:30.050 --> 00:25:34.530
selecting content on the screen. Um, so screen scraping a a lot of those things,

413
00:25:34.890 --> 00:25:36.730
anything from interacting to stealing content.

414
00:25:37.470 --> 00:25:40.890
Got it. Thank you, Chris. Pete, I see you have your hands up.

415
00:25:41.360 --> 00:25:44.170
Yeah, I just wanted to add that one other element to this is, uh,

416
00:25:44.350 --> 00:25:46.610
is the element of fairness. Uh, it's,

417
00:25:46.640 --> 00:25:51.050
it's crucial to have policies and procedures in place and to follow those. So,

418
00:25:51.110 --> 00:25:55.810
uh, if candidate A behaves in one way and ends up getting, uh,

419
00:25:55.810 --> 00:26:00.090
getting banned from the program and candidate B just gets a warning, uh, that,

420
00:26:00.090 --> 00:26:04.250
that news travels very quickly and is very damaging to a program. So, uh,

421
00:26:05.270 --> 00:26:08.250
the other side of it is one of the worst things that can happen is to, uh,

422
00:26:08.700 --> 00:26:13.490
invalidate or ban someone from a program incorrectly. Those accidents happen.

423
00:26:14.110 --> 00:26:17.650
Uh, so a lot of programs, including ours,

424
00:26:17.680 --> 00:26:21.770
have appeals processes, uh, at every stage so that, uh,

425
00:26:22.030 --> 00:26:26.810
in the one in a million chance that someone is incorrectly invalidated or banned

426
00:26:26.810 --> 00:26:29.850
from a program, there's the opportunity to go back, take a second look,

427
00:26:29.850 --> 00:26:33.490
and make sure that that was done correctly, so that nobody is, nobody is harmed.

428
00:26:33.750 --> 00:26:35.670
Mm-hmm. Yeah, I didn't,

429
00:26:35.670 --> 00:26:38.270
I've never thought about that before the appeals process, how many, um,

430
00:26:38.570 --> 00:26:41.430
you said appeals process at every stage, sort of how many,

431
00:26:41.570 --> 00:26:42.590
how many stages is that?

432
00:26:44.060 --> 00:26:46.550
Well, it depends on what the violation is. So if it's a,

433
00:26:46.550 --> 00:26:51.030
if it's a data forensics related, uh, invalidation, we have two different, uh,

434
00:26:51.130 --> 00:26:53.310
two different appeals processes. Uh,

435
00:26:53.310 --> 00:26:57.030
otherwise if it's a invalidation or a ban from the program, we have, uh,

436
00:26:57.090 --> 00:27:01.270
we have one, uh, that typically involves us going back and reviewing video,

437
00:27:01.460 --> 00:27:06.230
reviewing, uh, score similarities, uh, and not only looking at the,

438
00:27:06.450 --> 00:27:07.710
at the exam in question,

439
00:27:07.710 --> 00:27:11.190
but looking at any other exam that candidate has ever taken with us. Mm-hmm.

440
00:27:11.270 --> 00:27:14.870
So if we can discover a pattern from six months or a year ago that, uh,

441
00:27:14.870 --> 00:27:17.110
that carried forward into a, uh, suspicious exam,

442
00:27:17.690 --> 00:27:21.190
all of that is fair game for our, uh, our appeals investigations.

443
00:27:22.030 --> 00:27:26.520
Hmm. That's good to know. All right. Well, thank you so much. Uh,

444
00:27:26.590 --> 00:27:29.200
next question in this area, I'm curious,

445
00:27:29.420 --> 00:27:33.720
and this one I kind of wanna go to all panelists because I am sure you all have

446
00:27:33.990 --> 00:27:36.280
some fun stories to share with us. Um,

447
00:27:37.530 --> 00:27:40.480
would like to know some approaches that some, you know,

448
00:27:40.620 --> 00:27:45.320
bad actors or misbehaviors, uh, you know, would take to,

449
00:27:46.100 --> 00:27:48.720
to misbehave on a test or, or steal content, you know,

450
00:27:48.720 --> 00:27:51.960
what are some things that you've seen in terms of the range of ways that people

451
00:27:52.100 --> 00:27:56.960
can, can misbehave? Let's start with, um, Harry, and,

452
00:27:56.980 --> 00:27:58.000
and then we'll go around.

453
00:27:59.060 --> 00:28:03.880
So, so what I'm about to say comes from, you know, the experience of,

454
00:28:04.380 --> 00:28:08.760
um, my team investigating this for 500 or so clients. And so,

455
00:28:09.220 --> 00:28:09.620
um,

456
00:28:09.620 --> 00:28:13.320
the way I I start this description always in presentations is there's a million

457
00:28:13.320 --> 00:28:14.560
stories in the naked city.

458
00:28:14.560 --> 00:28:19.120
There's lots of different ways that bad guys think to threaten

459
00:28:19.350 --> 00:28:22.320
exam content. Um, however, that being said,

460
00:28:22.380 --> 00:28:26.840
you can divide those really only into two categories, and it's important to,

461
00:28:26.840 --> 00:28:30.240
to simplify it when you're thinking about it. So, for all the different schemes,

462
00:28:30.860 --> 00:28:33.680
really the only two things there are are proxy testing,

463
00:28:33.770 --> 00:28:37.360
which is another person taking the exam for the benefit of the candidate,

464
00:28:37.940 --> 00:28:42.160
or content harvesting where someone is stealing the intellectual property of the

465
00:28:42.160 --> 00:28:44.760
exam sponsor so they can sell it or distribute it.

466
00:28:45.150 --> 00:28:49.480
Both are significant threats to the exam program. Um, but that are the two,

467
00:28:49.610 --> 00:28:53.920
those are the two broad general categories. Mm-hmm. And within, within that,

468
00:28:54.380 --> 00:28:56.240
you know, we see all sorts of different attacks.

469
00:28:56.300 --> 00:29:01.000
We see technical attacks where some pretty sophisticated folks, um,

470
00:29:01.340 --> 00:29:02.360
who have, you know,

471
00:29:02.880 --> 00:29:06.640
excellent knowledge of how locked secure browsers work and how, you know,

472
00:29:06.640 --> 00:29:10.080
tests are delivered, um, attempt to threaten those using technology.

473
00:29:10.500 --> 00:29:15.080
We also see simple, very simple, um, measures taken where, um,

474
00:29:15.420 --> 00:29:15.730
you know,

475
00:29:15.730 --> 00:29:19.840
folks understand the limitations of the webcam and the microphone on the,

476
00:29:19.840 --> 00:29:24.120
you know, native to the candidate system, and they take steps to avoid that. Um,

477
00:29:24.420 --> 00:29:27.200
you know, the techniques are really limited only by the,

478
00:29:27.260 --> 00:29:30.880
the bad guy's imagination. And they definitely,

479
00:29:30.880 --> 00:29:33.720
definitely respond to our countermeasures.

480
00:29:33.780 --> 00:29:38.320
So as we implement a new technique or a new exam security measure,

481
00:29:38.510 --> 00:29:42.120
they are already very quickly working on a new way to get around that.

482
00:29:42.220 --> 00:29:44.320
And so it's a, it's a constant back and forth

483
00:29:44.780 --> 00:29:48.040
Mm-hmm. Constantly evolving. Pete, you have your hand up.

484
00:29:48.160 --> 00:29:49.680
I think that means you wanna answer next,

485
00:29:52.760 --> 00:29:55.120
Actually, I meant I forgot to, to lower my hand, but I'll, uh,

486
00:29:55.120 --> 00:29:56.400
I'll give it a shot. Oh,

487
00:29:58.140 --> 00:29:59.280
Please go ahead. Uh,

488
00:29:59.580 --> 00:30:04.280
For individuals taking, uh, online proctored exams, the, the two most common,

489
00:30:04.820 --> 00:30:05.280
uh,

490
00:30:05.280 --> 00:30:09.880
misbehaviors that we witness are either a third party present in the room or

491
00:30:09.990 --> 00:30:13.000
accessing a cell phone. Uh, if they're accessing the cell phone mm-hmm.

492
00:30:13.080 --> 00:30:13.840
One of two things is happening.

493
00:30:13.840 --> 00:30:17.200
They're either researching answers for a test question,

494
00:30:17.660 --> 00:30:22.160
or they're harvesting questions by taking photographs with a, with a cell phone,

495
00:30:22.460 --> 00:30:24.080
uh, for a third party. Uh,

496
00:30:24.260 --> 00:30:28.520
it could be as simple as someone standing behind them and indicating this,

497
00:30:28.520 --> 00:30:32.680
this one is answer A, this is answer choice B, this is answer choice C. Mm-hmm.

498
00:30:32.760 --> 00:30:34.720
Uh, in some cases, a third party present that is, uh,

499
00:30:34.720 --> 00:30:37.640
that is operating the keyboard. Uh,

500
00:30:38.100 --> 00:30:41.680
or we see very frequently a combination of the two where a third party is

501
00:30:41.920 --> 00:30:46.040
actually photographing, uh, the screen and harvesting on harvesting content.

502
00:30:46.280 --> 00:30:50.360
I had one of those last week where the candidate was taking the exam,

503
00:30:50.500 --> 00:30:52.240
he had his arms spread aside,

504
00:30:52.620 --> 00:30:55.520
and all of a sudden his cell phone creeps up in the, uh,

505
00:30:55.520 --> 00:30:59.120
between his arm and his, uh, and his chest and took a picture. Uh,

506
00:30:59.280 --> 00:31:03.440
I noticed that 25 times and a 50 question exam. So, uh, either,

507
00:31:03.460 --> 00:31:07.120
either someone has three arms or there was a third party behind, uh,

508
00:31:07.120 --> 00:31:10.560
operating there. So that's what we see most frequently. Wow.

509
00:31:11.230 --> 00:31:15.560
That is, that is impressive. Um, Chris,

510
00:31:15.630 --> 00:31:20.200
I'll go to you next. Uh, anything else to add here about some, some, some, uh,

511
00:31:21.080 --> 00:31:25.080
I guess, innovative approaches that you've seen or heard of? Yeah.

512
00:31:25.380 --> 00:31:28.680
Um, yeah, so, so yeah, the, the hidden camera, um,

513
00:31:28.780 --> 00:31:30.960
didn't sound like that last person hit it too well,

514
00:31:31.020 --> 00:31:35.240
but the camera take taking images of the screen. I think another, another,

515
00:31:35.240 --> 00:31:38.280
another area. So, so you have the proxy testing, you have the steam,

516
00:31:38.500 --> 00:31:42.000
the screen scaling content. I also think that there's a, um,

517
00:31:42.070 --> 00:31:45.760
kind of a self-help cheating too. And so what I mean by that is,

518
00:31:45.780 --> 00:31:47.840
is figuring out a way to beat the,

519
00:31:47.940 --> 00:31:51.600
the security measures in place in order to just help yourself take the exam.

520
00:31:52.060 --> 00:31:56.120
And so an example of that would be you figuring out how to use a virtual machine

521
00:31:56.180 --> 00:31:59.840
to, to get around the secure browser so that way you can, um,

522
00:31:59.980 --> 00:32:01.680
use Google yourself during the exam.

523
00:32:01.700 --> 00:32:03.720
So similar like what Pete said with using a phone,

524
00:32:03.740 --> 00:32:05.480
but doing it right on the screen so it's a little,

525
00:32:05.580 --> 00:32:09.800
little less detectable and working around those ways. I think, uh,

526
00:32:10.000 --> 00:32:14.360
GPT is really kind of expanding that 'cause it's, while,

527
00:32:14.360 --> 00:32:16.440
while Google you can eventually get to the answer,

528
00:32:16.440 --> 00:32:18.200
your hands are gonna have to do a lot of movement,

529
00:32:18.200 --> 00:32:21.480
and it's gonna be obvious to a proctor that you're doing that. Whereas GPT,

530
00:32:21.830 --> 00:32:25.840
it's a lot, you have to do a lot less to get a much better or accurate answer.

531
00:32:25.900 --> 00:32:29.840
And then depending on what kind of attacks there are, they could be done as,

532
00:32:29.840 --> 00:32:34.120
as simple as a, a hot stroke or a hotkey, um, implementation depending on what,

533
00:32:34.430 --> 00:32:35.400
what, what the, uh,

534
00:32:35.790 --> 00:32:38.760
what the actor can figure out what's allowed and what's not allowed. So,

535
00:32:39.140 --> 00:32:41.680
so when I say GPTT, I'm talking about like Chat GPT,

536
00:32:41.680 --> 00:32:44.000
those LLMs that are out there and you asking a question,

537
00:32:44.020 --> 00:32:47.080
it gives you the answer. You can even put in the whole multiple choice question.

538
00:32:47.220 --> 00:32:50.240
So, so figuring out those ways and, and combining that,

539
00:32:50.430 --> 00:32:53.080
combining those attacks between content stealing,

540
00:32:53.180 --> 00:32:56.320
and then just taking that content and moving it right off to the l l m,

541
00:32:56.500 --> 00:33:00.200
the chat GPT and getting your answer back and, and doing those type of things.

542
00:33:00.200 --> 00:33:03.360
So I think self-help, um, I'm referring to it as self-help,

543
00:33:03.360 --> 00:33:07.640
but basically a giving yourself a way to cheat without needing to evolve a third

544
00:33:07.640 --> 00:33:11.720
party is also, um, is also something I've seen out there. Hmm.

545
00:33:13.310 --> 00:33:17.480
Alright. Thank you. Uh, definitely lots of different perspectives. Jon,

546
00:33:17.840 --> 00:33:20.960
anything to, to round us out with this laundry list?

547
00:33:22.860 --> 00:33:26.490
Definitely, and there was a, a question in Q and A that I, uh,

548
00:33:26.850 --> 00:33:30.170
answered about Chat GTP and asking if it was something that had really, um,

549
00:33:30.400 --> 00:33:32.930
created more incident of cheating. And, um,

550
00:33:33.090 --> 00:33:34.650
I appreciate what you said about that, Chris. Like,

551
00:33:34.790 --> 00:33:38.650
people are finding better and better ways to get information out of the exam

552
00:33:39.200 --> 00:33:40.610
into any type of a,

553
00:33:40.750 --> 00:33:43.850
an AI model that'll help answer the question and get it back into the exam and

554
00:33:43.850 --> 00:33:44.810
it's getting faster and faster.

555
00:33:44.910 --> 00:33:48.450
So instead of people trying to google the answers they're using, um,

556
00:33:48.730 --> 00:33:51.130
a model like that, which is definitely it,

557
00:33:51.130 --> 00:33:55.610
it's not really increasing the number of, um, times that people attempt, um,

558
00:33:55.750 --> 00:33:57.410
to bypass the rules and look up answers,

559
00:33:57.410 --> 00:33:59.490
but it does definitely give them a better tool.

560
00:33:59.490 --> 00:34:01.730
It's more likely to give them something accurate as an answer,

561
00:34:01.980 --> 00:34:06.170
which is definitely a problem. Um, from our side as like our,

562
00:34:06.230 --> 00:34:08.330
our proctor team seems everything, like,

563
00:34:08.330 --> 00:34:11.090
there's funny stories about moms hiding under desks, you know,

564
00:34:11.380 --> 00:34:13.850
which gets discovered in the room pan to try and, you know,

565
00:34:13.850 --> 00:34:15.010
whisper the answers to their,

566
00:34:15.700 --> 00:34:19.490
their kids to all sorts of shenanigans like that. Um,

567
00:34:20.000 --> 00:34:24.570
sometimes it takes proctors, um, communicating amongst the team and being like,

568
00:34:24.790 --> 00:34:29.410
Hey, we've seen the same background a few too many times and identifying proxy

569
00:34:29.410 --> 00:34:31.770
testing rings based on their location, um,

570
00:34:31.770 --> 00:34:34.930
because there's not that many cubicles that color on a given day.

571
00:34:36.070 --> 00:34:38.890
And so we definitely find that, like Harry said,

572
00:34:38.890 --> 00:34:40.330
it's something that's continually evolving.

573
00:34:40.330 --> 00:34:45.050
People are always trying to find a new and innovative way to circumvent the

574
00:34:45.050 --> 00:34:45.710
rules. Um,

575
00:34:45.710 --> 00:34:49.930
and one thing that's so different about testing online is it's a lot more

576
00:34:50.130 --> 00:34:50.963
scalable. Um,

577
00:34:51.120 --> 00:34:54.850
it's not that scalable to physically go to a test center and take a test for

578
00:34:55.010 --> 00:34:57.530
somebody, but it's much more accessible now, um,

579
00:34:57.550 --> 00:35:00.610
in the same way that our tests are more accessible now, which has good and bad.

580
00:35:00.610 --> 00:35:01.110
Mm-hmm.

581
00:35:01.110 --> 00:35:03.010
Mm-hmm. Wow.

582
00:35:03.090 --> 00:35:06.490
I think we've just taught the audience a lot about some of the different ways

583
00:35:06.490 --> 00:35:11.450
that people, uh, uh, try to misbehave. So,

584
00:35:11.500 --> 00:35:11.850
Chris,

585
00:35:11.850 --> 00:35:15.450
there's one thing that you mentioned earlier that I wanna come back to a little

586
00:35:15.450 --> 00:35:16.370
bit. You mentioned,

587
00:35:16.410 --> 00:35:21.130
I think in your first answer that a secure browser is not enough. So, you know,

588
00:35:21.710 --> 00:35:24.370
can you tell me a little bit more about some of the various tools or

589
00:35:24.370 --> 00:35:27.570
technologies in the administrator's toolbox and,

590
00:35:27.590 --> 00:35:30.730
and why is a secure browser not enough at minimum?

591
00:35:32.550 --> 00:35:36.450
Um, yeah, so, so yeah, I, I definitely do not think a secure browser is,

592
00:35:36.510 --> 00:35:39.400
is enough. There's a lot of layers that need to be put in when you're,

593
00:35:39.400 --> 00:35:41.920
when you're, um, putting together a security implementation. I,

594
00:35:42.160 --> 00:35:44.640
I do really like the two camera option. I also,

595
00:35:44.720 --> 00:35:46.720
I recognize it doesn't make sense for everybody. It,

596
00:35:46.720 --> 00:35:48.360
it adds more friction in certain points.

597
00:35:48.420 --> 00:35:50.040
So it's not something you always want to use,

598
00:35:50.040 --> 00:35:53.240
but it is a useful tool if it makes sense for you and it's something that can

599
00:35:53.240 --> 00:35:55.000
work for your program. Um,

600
00:35:55.000 --> 00:35:57.880
there's different levels of proctoring too that can be implemented.

601
00:35:57.880 --> 00:36:01.600
So you can have like live real-time proctors, professional proctors, um,

602
00:36:02.060 --> 00:36:05.720
or more with like the, the Proctor Now solutions you can have like,

603
00:36:05.720 --> 00:36:09.320
you bring your own proctor or like a record and review or a hybrid model where

604
00:36:09.320 --> 00:36:12.040
really the candidate doesn't really know if they're being live proctor or not,

605
00:36:12.040 --> 00:36:15.680
but you have the tools in place and you can come and review after the fact. And,

606
00:36:15.700 --> 00:36:18.760
and AI notifications are also a useful tool too, um,

607
00:36:18.760 --> 00:36:22.280
with regards to proctoring and helping out and just improving the efficiency of

608
00:36:22.280 --> 00:36:26.200
the proctor. Um, secure browsers are definitely interesting.

609
00:36:26.200 --> 00:36:28.080
There's a whole bunch of different ones out there.

610
00:36:28.560 --> 00:36:30.560
I think when with regards to remote proctoring,

611
00:36:30.560 --> 00:36:33.520
you really need to use an application based lockdown browser.

612
00:36:33.660 --> 00:36:35.960
And so when I say secure browser, a lockdown browser,

613
00:36:36.350 --> 00:36:39.840
what I'm talking about is an application that can lock down the machine and only

614
00:36:39.890 --> 00:36:44.560
allow access to the applications and the content that should be taken during the

615
00:36:44.560 --> 00:36:48.520
exam. So that way you can't use other things during the exam. Um,

616
00:36:48.520 --> 00:36:53.130
there's different levels of it, um, but really it, it's, it's,

617
00:36:53.160 --> 00:36:56.210
it's a program at the end of the day, and so there,

618
00:36:57.050 --> 00:36:59.970
a secure browser in my opinion, can be beat.

619
00:37:00.030 --> 00:37:03.610
If there's somebody determined enough out there and they have the time and the

620
00:37:03.850 --> 00:37:07.130
resources and the motivation to do it, they, they will find a way.

621
00:37:07.130 --> 00:37:08.130
And so it's something that,

622
00:37:08.130 --> 00:37:11.850
it's an area that you constantly have to be vigilant in, constantly have to be,

623
00:37:11.910 --> 00:37:16.770
um, watching for, for new attacks out there and providing the updates. Um,

624
00:37:17.030 --> 00:37:17.863
the secure browser,

625
00:37:18.000 --> 00:37:20.810
it's so much more than just looking for processes these days.

626
00:37:21.110 --> 00:37:22.770
It used to be housed behind, uh,

627
00:37:23.080 --> 00:37:26.770
testing center with trusted admins and those type of things. But with,

628
00:37:26.770 --> 00:37:28.850
in the remote world, those days are long gone.

629
00:37:28.910 --> 00:37:30.930
People now have access to the software,

630
00:37:30.930 --> 00:37:33.730
they can download it because they need to. And so you need to,

631
00:37:33.730 --> 00:37:36.290
you need to take extra steps that didn't have to be taken before.

632
00:37:36.290 --> 00:37:38.330
You need to make sure that these, um,

633
00:37:38.480 --> 00:37:40.450
attackers trying to use the secure browser,

634
00:37:40.510 --> 00:37:43.210
you can't get like unlimited attempts to try to figure out a solution.

635
00:37:43.210 --> 00:37:46.170
You need to make sure that you're applying security in a smart way.

636
00:37:46.170 --> 00:37:48.810
You need to make sure that you're not only looking for processes,

637
00:37:48.830 --> 00:37:49.930
but you're doing for, you're,

638
00:37:49.930 --> 00:37:54.250
you're looking at the operating system holistically are, are every,

639
00:37:54.250 --> 00:37:58.570
every API and every interface that the secure browser's using is something

640
00:37:58.570 --> 00:38:02.930
else another application can do. Um, so you, you need to monitor these things.

641
00:38:02.930 --> 00:38:06.290
You need to, you need to watch for process hijacking and those type of things.

642
00:38:06.430 --> 00:38:10.130
Um, so it's, it's really just the gamut. And at the end of the day, uh, all,

643
00:38:10.130 --> 00:38:11.730
when you consider all those things together,

644
00:38:12.150 --> 00:38:15.690
it just makes it so secure browser is a very good tool and something you should

645
00:38:15.690 --> 00:38:18.930
have, but it should not be the only tool in the toolbox. Mm-hmm.

646
00:38:19.780 --> 00:38:22.010
Great. Thanks. Jon. Would you like to add?

647
00:38:23.440 --> 00:38:24.530
Yeah, it's interesting.

648
00:38:24.530 --> 00:38:28.170
There's two questions in the Q and A that really kind of tie into this one

649
00:38:28.170 --> 00:38:31.250
asking about how do you totally secure an environment and then the other,

650
00:38:31.710 --> 00:38:34.530
how effective secure browsers are detecting processes.

651
00:38:35.060 --> 00:38:38.960
And I think the important thing to realize is there's no complete single

652
00:38:39.320 --> 00:38:42.040
solution. We can never perfectly secure an environment.

653
00:38:42.470 --> 00:38:46.680
What we have to do is go back to that original plan and strategy for your exam

654
00:38:46.820 --> 00:38:50.170
and determine how much risk, um, you can afford,

655
00:38:50.710 --> 00:38:53.690
what's the right way based on the security, or sorry,

656
00:38:53.770 --> 00:38:55.050
based on the exam and your goals,

657
00:38:55.050 --> 00:38:58.530
what's the right level of security to introduce the right amount of friction to

658
00:38:58.530 --> 00:39:00.730
balance your security and user experience goals.

659
00:39:01.190 --> 00:39:02.810
So there isn't perfect solutions,

660
00:39:02.830 --> 00:39:07.090
and it's always going to be an iterative process as we introduce new

661
00:39:07.090 --> 00:39:09.130
countermeasures to, Harry said this exactly right. Like,

662
00:39:09.140 --> 00:39:11.610
we'll introduce a new countermeasure, there'll be a new response to it,

663
00:39:11.950 --> 00:39:13.410
and that's gonna be continuous forever.

664
00:39:14.190 --> 00:39:17.370
The key is that you set goals for your program that makes sense,

665
00:39:17.870 --> 00:39:20.290
so that you know how you're gonna respond if it happens.

666
00:39:20.670 --> 00:39:24.330
You understand that you put the right kind of set of protections into place

667
00:39:24.340 --> 00:39:27.130
based on the significance of the exam, um,

668
00:39:27.390 --> 00:39:31.170
and do your best to monitor the data that comes back so you can make adjustments

669
00:39:31.170 --> 00:39:31.910
over time.

670
00:39:31.910 --> 00:39:35.630
Mm-hmm. Great. Thank you. Um,

671
00:39:35.890 --> 00:39:40.510
so last question about, uh, that pertains to sort of during the exam here.

672
00:39:40.930 --> 00:39:44.390
Uh, we mentioned this a little bit earlier about some of the distractions, uh,

673
00:39:44.390 --> 00:39:47.670
coming in the room and some things that are a little bit out of your control,

674
00:39:47.760 --> 00:39:50.110
right? Some things may be common like someone,

675
00:39:50.430 --> 00:39:54.030
a child or a pet or something entering the room. But, you know,

676
00:39:54.030 --> 00:39:56.910
curious from your perspective, what are,

677
00:39:56.930 --> 00:40:00.750
what's sort of like that allowable threshold for those distractions? Uh,

678
00:40:00.750 --> 00:40:02.430
this kind of plays into that, you know,

679
00:40:02.430 --> 00:40:04.750
user friendliness versus security as well, you know,

680
00:40:04.970 --> 00:40:07.430
how do you make that determination? Um, Jon,

681
00:40:07.870 --> 00:40:12.130
anything to add to that or any, any thoughts on that question?

682
00:40:13.960 --> 00:40:16.060
Um, I got distracted reading one of the questions

683
00:40:16.640 --> 00:40:19.700
To ask again. I'm sorry. That's okay. No, it's okay. We talked about, you know,

684
00:40:19.700 --> 00:40:22.940
the candidate controls the environment, distractions may be common. Um,

685
00:40:23.200 --> 00:40:23.460
you know,

686
00:40:23.460 --> 00:40:27.020
what's the allowable threshold for allowing those distractions before an exam's

687
00:40:27.180 --> 00:40:28.013
terminated?

688
00:40:28.190 --> 00:40:29.023
Terminated? Yeah.

689
00:40:29.580 --> 00:40:32.780
I think that there needs to be a really specific choice made by the program

690
00:40:32.990 --> 00:40:34.260
ahead of time. Um,

691
00:40:34.720 --> 00:40:39.540
and what I mean by that is you need to think about how high stakes the

692
00:40:39.540 --> 00:40:43.620
exam is. And we generally approach that with, um,

693
00:40:43.730 --> 00:40:47.540
kind of packaging it in two general default things of is this something that is,

694
00:40:47.760 --> 00:40:51.660
um, a strict exam or a more balanced exam?

695
00:40:52.240 --> 00:40:56.380
And what we will do with that is determine what threshold it is where,

696
00:40:56.920 --> 00:40:58.940
for example, if a child walks into the room,

697
00:40:58.940 --> 00:41:02.540
there are certain programs that choose to terminate an exam at that point

698
00:41:02.540 --> 00:41:04.100
because of that violation.

699
00:41:04.360 --> 00:41:08.740
And so our goal is to have a suite of tools that a program can use to customize

700
00:41:09.170 --> 00:41:14.020
what level they want, um, based on their needs. So from our perspective,

701
00:41:14.080 --> 00:41:16.780
we have to have a, a wide range of that,

702
00:41:16.800 --> 00:41:20.260
but it's really down to the program to determine what's most appropriate. Um,

703
00:41:20.760 --> 00:41:24.140
and there's definitely not a single size that fits all. Mm-hmm.

704
00:41:25.790 --> 00:41:27.370
Um, Pete, anything to add to that?

705
00:41:28.130 --> 00:41:31.220
Yeah, to add to that, uh, it has to be iterative. Uh,

706
00:41:31.400 --> 00:41:33.480
at Amazon we used to have a very hard line.

707
00:41:33.740 --> 00:41:37.960
Any third party in the testing area equaled, uh,

708
00:41:38.120 --> 00:41:40.760
invalidation of the exam and termination of the exam.

709
00:41:41.260 --> 00:41:45.120
And then we realized that there are some circumstances where a toddler walks in

710
00:41:45.120 --> 00:41:47.760
to say hi to daddy or mom, uh,

711
00:41:48.300 --> 00:41:52.480
or you've got somebody in the, in the far background that opens the door,

712
00:41:52.640 --> 00:41:55.920
realizes that somebody's taking an exam and leaves, uh,

713
00:41:55.920 --> 00:42:00.840
without interacting whatsoever with a candidate. It was grossly unfair to, uh,

714
00:42:00.940 --> 00:42:04.720
to just automatically terminate those exams and require somebody to pay a fee

715
00:42:04.720 --> 00:42:08.800
for testing again. So, um, mm-hmm. We, we back down on that a little bit.

716
00:42:08.980 --> 00:42:12.840
And obviously if a child enters, we're assuming that they're not giving any, uh,

717
00:42:13.020 --> 00:42:17.560
any pre-knowledge of test questions. Uh, but, uh, but also, uh,

718
00:42:18.100 --> 00:42:22.960
we, we provide two warnings for that before we, uh, before we terminate an exam.

719
00:42:23.140 --> 00:42:23.973
Mm-hmm.

720
00:42:24.230 --> 00:42:29.030
Well, stricter balanced exam, you know,

721
00:42:29.030 --> 00:42:29.990
needing it to be iterative.

722
00:42:30.070 --> 00:42:31.910
I think those all sound like really good approaches to,

723
00:42:31.930 --> 00:42:35.000
to make that decision for your program. All right.

724
00:42:35.000 --> 00:42:36.840
So let's move into the final sort of phase here,

725
00:42:36.840 --> 00:42:40.840
talking about what you should be doing after the testing happens. So, uh,

726
00:42:40.900 --> 00:42:43.240
I'm gonna kick it off, Pete, I know you just talked, but, you know,

727
00:42:43.260 --> 00:42:45.160
I'm curious for your thoughts on this one. You know,

728
00:42:45.550 --> 00:42:50.200
tell me about what post-analysis steps should every testing

729
00:42:50.200 --> 00:42:51.080
program be doing?

730
00:42:52.280 --> 00:42:53.190
Thank you. Uh,

731
00:42:53.470 --> 00:42:58.150
I really think that that all depends on how well funded your exam program is.

732
00:42:58.290 --> 00:43:02.870
If you have a relatively small scope program and you're presenting 500 or a

733
00:43:03.070 --> 00:43:04.550
thousand exams a year, uh,

734
00:43:04.610 --> 00:43:09.110
you may not have funding to do things like full-blown data forensics, uh,

735
00:43:09.330 --> 00:43:14.070
and statistical analysis. Uh, in that case, just, uh, you know, some,

736
00:43:14.100 --> 00:43:18.790
some basic, uh, basic score analysis. Uh, so looking at it, uh,

737
00:43:18.890 --> 00:43:21.230
if, if the typical candidate for this, uh,

738
00:43:21.230 --> 00:43:24.510
this particular exam takes 45 minutes to complete an exam,

739
00:43:24.850 --> 00:43:28.510
and you have somebody that completed it in six minutes and 30 seconds, uh,

740
00:43:28.580 --> 00:43:31.550
that might be a red flag that this is something worth looking at. Um,

741
00:43:31.770 --> 00:43:35.390
if you're well funded, you have a, uh, a higher scope program, and you're, uh,

742
00:43:35.410 --> 00:43:39.830
you're delivering tens of thousands or hundreds of thousands of exams a year,

743
00:43:40.330 --> 00:43:44.590
uh, then you may have the, uh, the, the fortune of, uh, of, uh,

744
00:43:45.610 --> 00:43:49.630
of, uh, having a team that can look at, uh, data forensics,

745
00:43:49.630 --> 00:43:52.950
looking at score similarities, looking at, uh, at all of, uh,

746
00:43:53.210 --> 00:43:56.990
all those different elements to, uh, to determine whether or not, uh,

747
00:43:56.990 --> 00:44:00.920
whether or not misbehavior is occurring that way. Mm-hmm.

748
00:44:02.550 --> 00:44:04.460
Harry, wonder if you have anything to add to that?

749
00:44:05.460 --> 00:44:08.580
I do. Um, just to quote another one of our clients, and,

750
00:44:08.580 --> 00:44:11.260
and that's something I'll put in a pitch for here. Um,

751
00:44:11.400 --> 00:44:13.660
we conduct investigations jointly with our clients.

752
00:44:13.800 --> 00:44:18.180
We learn from them and they learn from us. And so, um, when I get to this later,

753
00:44:18.180 --> 00:44:19.500
I'll talk about joint investigations.

754
00:44:19.580 --> 00:44:21.980
I think another question is more appropriate to that,

755
00:44:22.040 --> 00:44:23.780
but the reason I'm quoting my,

756
00:44:23.840 --> 00:44:27.540
our clients so much is because we learn from them. And, and so this quote is,

757
00:44:28.400 --> 00:44:30.060
you really cannot, um,

758
00:44:30.420 --> 00:44:34.240
abdicate responsibility for the security of your own program. And, and by that,

759
00:44:34.240 --> 00:44:38.800
what this client meant is somebody at the exam sponsor has to be looking at

760
00:44:38.800 --> 00:44:42.440
results. And, and Pete, I think did a good job of explaining, you know,

761
00:44:42.440 --> 00:44:43.520
based on resources,

762
00:44:43.520 --> 00:44:46.200
maybe it's a whole psychometrics team that can do forensics and,

763
00:44:46.260 --> 00:44:47.880
and very detailed analysis.

764
00:44:48.330 --> 00:44:53.260
Maybe it's just one person based on the size of the program who has

765
00:44:53.260 --> 00:44:57.300
other roles and responsibilities, but who also has as a,

766
00:44:57.360 --> 00:44:59.620
as one of their collateral duties security.

767
00:45:00.040 --> 00:45:04.260
But somebody has to be flying the air, the security airplane, um,

768
00:45:04.320 --> 00:45:07.540
and monitoring it. And sadly, we do see clients where, you know,

769
00:45:07.590 --> 00:45:12.380
three or four people share that responsibility. And inevitably that's,

770
00:45:12.380 --> 00:45:17.180
those are the programs where sometimes the ball is dropped or, um, you know,

771
00:45:17.240 --> 00:45:19.860
no one's watching for periods of months or even years.

772
00:45:20.360 --> 00:45:24.700
And so it's important whatever level of scrutiny that you do that someone does

773
00:45:24.700 --> 00:45:27.740
it. And I will also add one other technique, um,

774
00:45:27.840 --> 00:45:32.460
in addition to forensics is a tip line, um, to, again, to quote a client,

775
00:45:32.520 --> 00:45:34.420
if you don't have a tip line, uh,

776
00:45:34.420 --> 00:45:37.660
then you are blind to the exam security threats that your program faces.

777
00:45:37.840 --> 00:45:39.820
And so it's important that you have a capability,

778
00:45:39.840 --> 00:45:42.220
and it doesn't have to be a man telephone, it can be an email,

779
00:45:42.800 --> 00:45:45.380
but your candidates are out there seeing what,

780
00:45:45.410 --> 00:45:47.660
what their colleagues are doing and what other people in their,

781
00:45:47.680 --> 00:45:49.580
in their position are doing. Um,

782
00:45:49.710 --> 00:45:54.220
oftentimes they wanna report misconduct themselves because of course they share

783
00:45:54.250 --> 00:45:57.820
your interest in keeping the program safe. The honest candidates always do.

784
00:45:58.280 --> 00:46:01.420
And so in addition to forensics and looking at scores,

785
00:46:01.520 --> 00:46:05.060
you also have to have a way for someone outside the,

786
00:46:05.160 --> 00:46:09.020
the company to be able to report to you if they see something bad happening. So,

787
00:46:09.020 --> 00:46:09.853
tip line. Mm-hmm.

788
00:46:10.040 --> 00:46:14.000
Tip line. Okay. Uh, we've already touched this a little bit,

789
00:46:14.060 --> 00:46:16.520
but I do wanna just make sure that we call out, you know,

790
00:46:16.870 --> 00:46:20.080
what is the balance between failing someone or allowing them to pass?

791
00:46:20.220 --> 00:46:24.280
If you think that they are misbehaving, you know, where do you draw that line?

792
00:46:24.300 --> 00:46:26.880
And I, I, I assume that we're going to say it sort of depends,

793
00:46:26.980 --> 00:46:30.360
but kind of curious what some of those decision factors are. Um, Jon,

794
00:46:30.420 --> 00:46:32.200
I'm wondering if you could say a few words on that.

795
00:46:35.420 --> 00:46:39.790
There's definitely a few, um, key factors. I, I think, um,

796
00:46:40.890 --> 00:46:45.230
really thinking about the way to secure, um, the environment,

797
00:46:45.250 --> 00:46:49.670
the pieces that, that come into that it's, it's again,

798
00:46:49.670 --> 00:46:53.210
putting the layers in place that you have to have, um,

799
00:46:54.270 --> 00:46:56.130
not leaving it until the moment, um,

800
00:46:56.320 --> 00:46:58.970
that you're just launching an exam or new version to make sure that you've

801
00:46:58.970 --> 00:47:02.890
thought about the continuums. Um, and there was a, a question in the qa, uh,

802
00:47:02.890 --> 00:47:06.520
well, about the, what layers go into the environment and it's,

803
00:47:07.230 --> 00:47:11.960
it's gotta be clear rules, clear responses to those rules. We typically rely,

804
00:47:12.180 --> 00:47:15.640
um, on tactics like room pads and other things,

805
00:47:15.640 --> 00:47:20.560
sometimes even partway through the exam to re-secure if there's an anomaly or

806
00:47:20.720 --> 00:47:24.640
something that might kind of cause us to question the situation. Um,

807
00:47:25.170 --> 00:47:27.320
those are kind of the top of mind for me. Mm-hmm.

808
00:47:28.270 --> 00:47:29.580
And Pete, anything to add to that?

809
00:47:30.490 --> 00:47:33.780
Well, I think it, it, uh, it also has something to do with, uh,

810
00:47:33.780 --> 00:47:37.260
with severity of offense and provability of offense.

811
00:47:37.840 --> 00:47:41.460
So if I were, if I wanted to scan, uh,

812
00:47:41.660 --> 00:47:45.860
a document that I have on my iPhone, I would place that outside of,

813
00:47:46.040 --> 00:47:47.340
of a webcam view.

814
00:47:47.960 --> 00:47:51.780
And the only way I'm gonna be able to access that is if I'm consistently looking

815
00:47:51.840 --> 00:47:55.060
to the side to see what's on, on that, uh,

816
00:47:55.720 --> 00:48:00.340
on that smartphone. Um, we use, in that case, we, uh,

817
00:48:00.520 --> 00:48:02.020
the first time somebody looks off screen,

818
00:48:02.020 --> 00:48:05.180
we issue a warning and let them know that continuing that can, uh,

819
00:48:05.200 --> 00:48:08.540
can result in termination of the exam. The second time we get a second warning,

820
00:48:08.670 --> 00:48:10.820
third time they do it, it's uh, it's invalidated.

821
00:48:10.840 --> 00:48:15.180
Now the proof side of that is we can't see from our webcam recording

822
00:48:15.650 --> 00:48:17.820
what exactly it is that they're looking at. We,

823
00:48:17.920 --> 00:48:20.060
we know it's a device or it's perhaps notes,

824
00:48:20.240 --> 00:48:23.740
but we don't know for certain that they are notes about the, uh, about the,

825
00:48:23.920 --> 00:48:28.260
the exam. Now, if somebody leaves webcam view, uh,

826
00:48:28.440 --> 00:48:29.160
for five,

827
00:48:29.160 --> 00:48:32.900
six seconds and then comes back or decides that they want a bathroom break,

828
00:48:32.900 --> 00:48:36.140
even though we don't offer those for our exams and comes back a minute and a

829
00:48:36.140 --> 00:48:40.380
half later, uh, that's a, that's an automatic, uh, violation and,

830
00:48:40.480 --> 00:48:43.300
and termination of the exam because, uh, you know,

831
00:48:43.320 --> 00:48:45.980
who knows what they were doing during that minute and a half if they were

832
00:48:45.980 --> 00:48:48.940
cribbing up on answers for a question that they're having a difficult time

833
00:48:48.940 --> 00:48:52.580
answering or, uh, or what the case might be. So it is, but, uh,

834
00:48:52.580 --> 00:48:55.940
provability does definitely play a, a role in this. Mm-hmm.

835
00:48:56.240 --> 00:48:59.660
And is it fair to say, Pete? A lot of it comes down to indicators,

836
00:48:59.690 --> 00:49:04.500
like we're looking for indicators of potential problems. And, um,

837
00:49:04.690 --> 00:49:08.500
it's, it's funny 'cause I think about this a lot. I, in my office, um,

838
00:49:08.570 --> 00:49:11.540
I've got a big window to my right and people walk their dogs by all the time,

839
00:49:12.000 --> 00:49:15.540
and I, uh, I'm always like, Ooh, you know, who's walking by? What's going on?

840
00:49:15.540 --> 00:49:15.740
Like,

841
00:49:15.740 --> 00:49:18.620
I look out there a lot and I think about if I was in this environment in a test,

842
00:49:19.640 --> 00:49:22.940
at what point in time would my natural habits get me in trouble with that?

843
00:49:23.000 --> 00:49:27.460
And so part of it is there may be a completely innocent reason,

844
00:49:27.480 --> 00:49:32.020
and that's why we really push on the way we flag and talk about events is

845
00:49:32.930 --> 00:49:36.180
what, like what's the indicators, what's the potential event? Like, we're,

846
00:49:36.180 --> 00:49:39.020
what we're trying to be very careful about how we capture that because we

847
00:49:39.220 --> 00:49:41.580
provide data to a program and so they can make a judgment call,

848
00:49:42.320 --> 00:49:45.260
but there might be an an excuse you never really know,

849
00:49:45.920 --> 00:49:50.260
but if there's enough indicators, chances are it might also not be innocent.

850
00:49:50.260 --> 00:49:53.660
And so you've gotta be watching and thinking about how to interpret that data

851
00:49:53.660 --> 00:49:55.700
because you can never have the full story. Mm-hmm.

852
00:49:57.000 --> 00:50:01.200
That's a good point. Alright, so Harry, uh,

853
00:50:01.810 --> 00:50:06.080
wanna chat with you for a a minute about major misconduct. This is one of,

854
00:50:06.110 --> 00:50:08.000
this is one of your major areas of focus.

855
00:50:08.460 --> 00:50:12.920
So what are some signs that major misconduct is afoot and it's not just a

856
00:50:12.920 --> 00:50:14.720
one-off, you know, and, uh,

857
00:50:15.110 --> 00:50:18.800
what are some indicators that a more in-depth investigation may be warranted?

858
00:50:19.700 --> 00:50:24.020
So I think, uh, I think, uh, we, we, the way I characterize,

859
00:50:24.080 --> 00:50:27.840
the way we do intake of investigations is, uh,

860
00:50:28.020 --> 00:50:30.000
we are not shy and we're not proud.

861
00:50:30.010 --> 00:50:33.680
We'll take information from anywhere as long as it's truthful and it's,

862
00:50:33.680 --> 00:50:36.000
it's accurate, it's verifiable. Um,

863
00:50:36.140 --> 00:50:39.160
so we will take lead information from major misconduct, from,

864
00:50:39.160 --> 00:50:40.800
from literally anyone, uh,

865
00:50:40.800 --> 00:50:43.960
as long as we can prove that the information is accurate. Um,

866
00:50:44.590 --> 00:50:47.160
sources that we will typically draw on.

867
00:50:47.160 --> 00:50:50.720
We have informants that we've recruited over time who are reporting to us what

868
00:50:50.720 --> 00:50:55.560
the bad guys are doing. Um, we, we use tip lines, both Pearson VUE has,

869
00:50:55.580 --> 00:50:59.840
we have our own tip line, and then our clients all almost all have, uh,

870
00:50:59.900 --> 00:51:03.960
robust tip line capabilities. And so, we'll, we'll take it that way. Um,

871
00:51:04.500 --> 00:51:07.080
we also, um, rely heavily on forensics, right?

872
00:51:07.080 --> 00:51:09.640
If our psychometric teams are telling us there's a problem,

873
00:51:09.940 --> 00:51:14.120
or because of joint investigations with our clients, our client's,

874
00:51:14.120 --> 00:51:17.320
psychometric teams are telling us, um, there's an issue,

875
00:51:17.320 --> 00:51:21.000
then we will look there. Um, and typically it doesn't start as a major problem,

876
00:51:21.000 --> 00:51:25.040
right? Uh, if, if one, you know, one person out there,

877
00:51:25.300 --> 00:51:27.120
one colleague of a candidate says, Hey,

878
00:51:27.120 --> 00:51:30.920
this guy sits in the next cubicle and I heard that he's cheating. Um, you know,

879
00:51:30.920 --> 00:51:34.080
I heard he, he had someone take his exam. Um, you know,

880
00:51:34.080 --> 00:51:37.480
we'll gather all the information from that tipster from that person and maybe

881
00:51:37.480 --> 00:51:41.440
it'll turn into a major investigation because we will recognize that the

882
00:51:41.680 --> 00:51:45.640
technique is one used by dozens or hundreds of candidates. Um, so again,

883
00:51:45.640 --> 00:51:48.720
it really depends where we get the information from. Um,

884
00:51:49.060 --> 00:51:51.080
we look at everything at least on the,

885
00:51:51.100 --> 00:51:54.880
on the VUE Special Investigations team as having the potential for being a

886
00:51:54.880 --> 00:51:59.520
large scale threat because oftentimes many of them are. Um,

887
00:51:59.580 --> 00:52:02.600
and then what we do with that information is also, I think, significant.

888
00:52:02.660 --> 00:52:04.360
So I'll just take a second to talk about it.

889
00:52:04.660 --> 00:52:08.120
In addition to sharing it with our clients so they can take remedial action and,

890
00:52:08.140 --> 00:52:10.520
and handle the candidate misconduct, um,

891
00:52:10.580 --> 00:52:13.640
we also share it with our internal teams. Um, you know,

892
00:52:13.640 --> 00:52:18.360
we are very aggressive at pushing techniques that we see, you know,

893
00:52:18.440 --> 00:52:23.120
what the bad guys are doing to our development teams so that they can implement

894
00:52:23.120 --> 00:52:27.040
their own countermeasures to stop them. And so we talk with the product owners,

895
00:52:27.140 --> 00:52:30.120
in this case, the on VUE product owners on a daily,

896
00:52:30.120 --> 00:52:33.600
sometimes hourly basis so they can build a higher wall.

897
00:52:34.060 --> 00:52:38.090
Mm-hmm. Uh, last question here. Uh,

898
00:52:38.670 --> 00:52:41.890
how often would you guys say a new technique pops up and,

899
00:52:41.910 --> 00:52:45.530
and how do you find out about it? Uh, I'll start with Chris quickly. Um,

900
00:52:45.590 --> 00:52:48.530
you know, how often would you say you're seeing new techniques to respond to?

901
00:52:49.730 --> 00:52:52.940
Yeah, I, I think it's more driven by just, um, what the market is, uh,

902
00:52:52.940 --> 00:52:57.340
how valuable that is and what, what workarounds have recent or what, um,

903
00:52:57.700 --> 00:53:00.340
security ha has the wall gotten higher recently. Mm-hmm. Um,

904
00:53:00.340 --> 00:53:02.700
every time the wall gets higher, it's gonna get, um,

905
00:53:02.700 --> 00:53:04.620
they're gonna come up with a new way to get around that.

906
00:53:04.680 --> 00:53:09.060
And so there's certainly, um, e even even at a very high,

907
00:53:09.060 --> 00:53:12.660
easy to level for some or easy level that somebody's not very exp experienced

908
00:53:12.660 --> 00:53:16.060
with computers, you can easily find a website that's out there selling,

909
00:53:16.060 --> 00:53:19.220
selling some of the attacks that I mentioned earlier, um, in,

910
00:53:19.220 --> 00:53:20.940
in the form of software. And so those,

911
00:53:20.940 --> 00:53:23.540
those things are constantly popping up every time a measure,

912
00:53:23.590 --> 00:53:25.920
every time you implement a measure, it,

913
00:53:25.940 --> 00:53:29.200
it pretty much you're starting a stopwatch for somebody to try to figure that

914
00:53:29.200 --> 00:53:31.680
out if there's value in, in figuring it out.

915
00:53:32.540 --> 00:53:34.800
Got it. Thank you, Harry, in a couple seconds,

916
00:53:35.200 --> 00:53:37.760
anything else to add to that about how, how often these pop up?

917
00:53:38.340 --> 00:53:42.200
No, it's, it's, I think Chris covered it. It's not super often, um, you know,

918
00:53:42.200 --> 00:53:46.800
the bad guys evolve, uh, for sure in response to what we're doing, but it, it,

919
00:53:46.800 --> 00:53:49.400
you know, it's not hour by hour, minute by minute, I mean,

920
00:53:49.400 --> 00:53:53.200
it takes them a while to figure out what we're doing and then to develop

921
00:53:53.560 --> 00:53:55.840
procedures. Um, you know, there's, there's been a,

922
00:53:55.840 --> 00:53:58.600
there was a big one earlier this year where they invented a whole new way of,

923
00:53:58.700 --> 00:54:00.960
of doing proxy testing. Um,

924
00:54:01.180 --> 00:54:05.840
but I'd say only one or two major new techniques a year, uh, by and large.

925
00:54:06.170 --> 00:54:07.880
Great. Alright,

926
00:54:08.030 --> 00:54:11.640
last presenter question before we grab some of the questions from the Q and A.

927
00:54:11.720 --> 00:54:14.600
I appreciate that you guys have been answering sort of in writing throughout,

928
00:54:14.980 --> 00:54:18.160
uh, in, as you know, just a couple of seconds I wanna go around the table.

929
00:54:18.250 --> 00:54:19.440
Where do you see us,

930
00:54:20.130 --> 00:54:23.240
where do you think we'll be in one to two years with regards to remote

931
00:54:23.240 --> 00:54:25.440
proctoring? Uh, Pete, I'm gonna start with you.

932
00:54:26.960 --> 00:54:28.400
I think, uh, I think both, uh,

933
00:54:28.400 --> 00:54:33.240
both proxy testers and misbehaving candidates and test delivery providers

934
00:54:33.580 --> 00:54:37.280
are gonna be using AI and machine learning a lot more than we're using now. Uh,

935
00:54:37.380 --> 00:54:38.800
the, the big, uh,

936
00:54:39.660 --> 00:54:43.960
the big detractors for both of those right now are things like GDPR and, uh,

937
00:54:43.960 --> 00:54:46.520
requirements and restrictions outside of the United States.

938
00:54:46.940 --> 00:54:50.880
And the fact that these programs are still in their infant stage, uh,

939
00:54:51.170 --> 00:54:54.200
don't work very well and are incredibly slow, but once, uh,

940
00:54:54.390 --> 00:54:59.200
once they speed up and are are real time as opposed to, uh, long pauses between,

941
00:54:59.620 --> 00:55:02.240
uh, between prompts and answers, I think we're gonna be in trouble.

942
00:55:04.300 --> 00:55:07.800
All right. Chris, you're next on my list. Anything, uh, any insights,

943
00:55:07.990 --> 00:55:09.040
predictions to add?

944
00:55:10.160 --> 00:55:11.160
I think, um, I,

945
00:55:11.280 --> 00:55:15.080
I do think that we'll see probably more record and review from a a proctoring

946
00:55:15.080 --> 00:55:18.520
standpoint, um, from a, from a bad actor standpoint,

947
00:55:18.680 --> 00:55:21.800
I think it will be the same but different. I think they'll, uh,

948
00:55:21.800 --> 00:55:23.280
they'll find new ways, but the,

949
00:55:23.300 --> 00:55:26.120
the same people trying to do it will still be trying to do it. If there're,

950
00:55:26.120 --> 00:55:29.080
if there's money there, they're, they're gonna be looking for that money. Uh,

951
00:55:29.080 --> 00:55:31.160
something I, something I am interested to see is,

952
00:55:31.180 --> 00:55:36.160
is does what is AI going to be used to help? Um,

953
00:55:36.230 --> 00:55:38.360
well, I, I'm, I know it is gonna be used, but like,

954
00:55:38.380 --> 00:55:41.760
how is AI gonna really make these, make them,

955
00:55:42.000 --> 00:55:44.040
I improve the frequency of their attack?

956
00:55:44.100 --> 00:55:48.120
Is it gonna increase the frequency of their attacks? Because AI, not only is it,

957
00:55:48.120 --> 00:55:51.560
uh, beneficial, but it also can be used to, um, make,

958
00:55:51.630 --> 00:55:54.440
make development and those type of things easier as well.

959
00:55:55.140 --> 00:55:58.840
Mm-hmm. Uh, Harry, anything to add? Any

960
00:55:58.840 --> 00:56:00.920
Prediction? Yeah, I would just, I would just say I wanted to,

961
00:56:00.920 --> 00:56:03.040
to add this earlier, but then we move past it.

962
00:56:03.620 --> 00:56:08.490
The real benefit of AI to the bad guys so far that we've been able to see

963
00:56:08.550 --> 00:56:11.570
is it reduces their need for experts. And what I mean by that is,

964
00:56:12.310 --> 00:56:14.290
if I set up a proxy, you know,

965
00:56:14.370 --> 00:56:19.210
a proxy business running it out of my house and I want to test on five different

966
00:56:19.240 --> 00:56:20.250
exam programs, you know,

967
00:56:20.250 --> 00:56:24.240
I wanna test a w s 'cause Pete's here and I want to test three or four other

968
00:56:24.270 --> 00:56:27.960
exam programs, chances are I can't, you know,

969
00:56:27.960 --> 00:56:32.200
Harry can't pass all of those exams himself. So he has to, you know,

970
00:56:32.280 --> 00:56:34.640
I have to go out and get experts. I have to get, you know,

971
00:56:34.860 --> 00:56:38.560
my wife can maybe pass an exam and then my next door neighbor can pass an exam.

972
00:56:38.940 --> 00:56:42.760
And then the coordination of scheduling all those folks and making those remote

973
00:56:42.760 --> 00:56:44.160
access proxy tests happen.

974
00:56:44.710 --> 00:56:49.120
What Chat GPT and the large learning models allow these proxy testers to do is

975
00:56:49.120 --> 00:56:52.480
dispense with those experts. Because as Chris described it,

976
00:56:52.500 --> 00:56:57.280
you can type in a portion of the question and Chat GPT will give you

977
00:56:58.000 --> 00:57:01.280
a reasonable answer that would let you take the, you know, answer the question.

978
00:57:01.700 --> 00:57:04.320
And so Chat GPT is really streamlining,

979
00:57:04.580 --> 00:57:09.040
at least has the potential to really streamline how these remote access proxy

980
00:57:09.190 --> 00:57:10.680
organizations do business.

981
00:57:12.770 --> 00:57:16.080
Definitely. All right. Jon, last prediction. I

982
00:57:16.950 --> 00:57:21.540
Think that you'll continue to see AI used more and more on both sides.

983
00:57:21.860 --> 00:57:23.140
Absolutely. And if you think about,

984
00:57:23.140 --> 00:57:25.060
especially from the certification perspective,

985
00:57:25.220 --> 00:57:27.940
a lot of what we're trying to do is help people demonstrate skills.

986
00:57:28.660 --> 00:57:33.460
I imagine we'll get to a point soon where AI is able to evaluate someone's

987
00:57:33.460 --> 00:57:34.040
performance.

988
00:57:34.040 --> 00:57:37.380
So if you think about performance-based testing and some of those things,

989
00:57:37.450 --> 00:57:39.740
it's a great way to demonstrate, like, it's one thing to say,

990
00:57:39.780 --> 00:57:43.260
I know the answers to an AWS exam. It's another thing to say,

991
00:57:43.300 --> 00:57:44.740
I know how to set up an e c two cluster.

992
00:57:45.440 --> 00:57:47.380
And one of the things that is gonna change,

993
00:57:47.540 --> 00:57:51.940
I think will be our ability to evaluate responses. So I,

994
00:57:52.180 --> 00:57:55.620
I would imagine that we'll start to see AI being able to evaluate our

995
00:57:55.620 --> 00:57:57.260
performance of a task, our response,

996
00:57:57.420 --> 00:58:00.420
something that's not just based on evaluating question and answers.

997
00:58:00.480 --> 00:58:04.860
And we've already seen it becoming much more able to understand big blocks of

998
00:58:05.060 --> 00:58:06.900
language, like essay answers and, and the like.

999
00:58:07.320 --> 00:58:10.500
So I think it's gonna be incredibly amazing from a learning perspective.

1000
00:58:10.940 --> 00:58:11.773
I think as humans,

1001
00:58:11.930 --> 00:58:16.100
it's gonna change the way that we access information in an extreme way,

1002
00:58:16.130 --> 00:58:19.340
just like the internet did and all of a sudden information's at our fingertips,

1003
00:58:19.650 --> 00:58:22.020
well now it'll be more digestible and more consumable.

1004
00:58:22.080 --> 00:58:25.340
So I think it's gonna be really exciting and there's gonna be endless amounts of

1005
00:58:25.340 --> 00:58:28.180
work for us to do, um, to keep testing secure and fair.

1006
00:58:28.240 --> 00:58:29.660
And I'm excited about it.

1007
00:58:30.810 --> 00:58:35.360
Great. Thank you. Oh my goodness. We are at time more or less,

1008
00:58:35.700 --> 00:58:40.200
um, I see in looking at the questions and answers that are here now,

1009
00:58:40.200 --> 00:58:42.360
like I said, thank you so much for, uh,

1010
00:58:42.360 --> 00:58:45.000
the panelists for answering those questions live. I,

1011
00:58:45.080 --> 00:58:48.560
I believe we can share the Q and A after we will have a sort of a,

1012
00:58:48.640 --> 00:58:53.070
a post webinar email going out. Um, uh, any,

1013
00:58:53.780 --> 00:58:56.270
I'll, I'll open it up to the panelists for, you know, the,

1014
00:58:56.270 --> 00:58:58.870
the most upvoted question in here is about, you know,

1015
00:58:58.870 --> 00:59:02.710
when someone's caught with misconduct, how much information is provided, uh,

1016
00:59:02.710 --> 00:59:04.070
and the details of their misconduct.

1017
00:59:04.220 --> 00:59:06.550
Does anyone wanna attempt an answer at that?

1018
00:59:10.900 --> 00:59:14.830
Maybe Pete or Jon, you just came off mute.

1019
00:59:15.810 --> 00:59:18.750
Uh, I was just gonna go on mute. I think probably Harry and Pete have the best,

1020
00:59:19.330 --> 00:59:21.880
we provide a lot of data, but we don't usually make those judgment calls.

1021
00:59:22.430 --> 00:59:24.760
Alright, Harry, since you came off mute, let us, uh,

1022
00:59:25.590 --> 00:59:29.600
Talk Yeah, yeah. To Jon's point, we never make those judgment calls. So, uh,

1023
00:59:29.640 --> 00:59:33.480
a famous quote from J Edgar Hoover, you know, the FBI is, you know,

1024
00:59:33.480 --> 00:59:35.880
the FBI is in fact gathering information. We don't,

1025
00:59:36.140 --> 00:59:40.760
we don't judge guilt or innocence. We provide the facts. That's exactly how, um,

1026
00:59:40.950 --> 00:59:43.560
this team operates. We provide, um,

1027
00:59:43.980 --> 00:59:46.960
all of the facts that are available to the client,

1028
00:59:47.100 --> 00:59:50.680
and then they compare them to their hundreds of different policies and

1029
00:59:50.880 --> 00:59:54.440
procedures and they make the decision on, you know, adjudication.

1030
00:59:54.440 --> 00:59:56.000
Sometimes they'll ask us what, you know,

1031
00:59:56.000 --> 00:59:58.840
what are other clients in the space doing or,

1032
00:59:58.900 --> 01:00:02.920
or what's an industry best practice? But, but we never make a decision. Um,

1033
01:00:02.980 --> 01:00:04.760
we will occasionally make recommendations,

1034
01:00:04.860 --> 01:00:09.560
but our primary role is providing facts to people like Pete and then they make

1035
01:00:09.560 --> 01:00:10.393
the decision.

1036
01:00:10.580 --> 01:00:15.390
Mm-hmm. All right. Well, thank you so much everybody. We are at time right now.

1037
01:00:15.950 --> 01:00:17.990
I, I wanna once again, thank you all for joining us.

1038
01:00:18.120 --> 01:00:20.870
Thank you so much to our panelists for enjoying us.

1039
01:00:21.030 --> 01:00:24.510
I think we all learned a lot. Um, like I said, uh,

1040
01:00:24.890 --> 01:00:27.110
you should be getting a follow-up email with a recording.

1041
01:00:27.440 --> 01:00:29.270
We'll see what we can do about the Q and A, uh,

1042
01:00:29.270 --> 01:00:31.310
the questions that have been answered. And, uh,

1043
01:00:31.320 --> 01:00:35.910
don't click away too fast for the survey. So thank you all once again.

1044
01:00:38.000 --> 01:00:38.833
Bye.

1045
01:00:38.970 --> 01:00:39.400
Thank you.