Meet “AI”, your new colleague: could it expose your company’s secrets? | WeLiveSecurity

Before
rushing
to
embrace
the
LLM-powered
“hire”,
make
sure
your
organization
has
safeguards
in
place
to
avoid
putting
its
business
and
customer
data
at
risk

Chatbots
powered
by
large
language
mod

Meet “AI”, your new colleague: could it expose your company’s secrets? | WeLiveSecurity

Before
rushing
to
embrace
the
LLM-powered
“hire”,
make
sure
your
organization
has
safeguards
in
place
to
avoid
putting
its
business
and
customer
data
at
risk

Chatbots
powered
by
large
language
models
(LLMs)
are
not
just
the
world’s
new
favorite
pastime.
The
technology
is
increasingly
being
recruited
to
boost
workers’
productivity
and
efficiency,
and
given
its
increasing
capabilities,
it’s
poised
to
replace
some
jobs
entirely,
including
in
areas
as
diverse
as
coding,
content
creation,
and
customer
service.

Many
companies
have
already
tapped
into
LLM
algorithms,
and
chances
are
good
that
yours
will
likely
follow
suit
in
the
near
future. In
other
words,
in
many
industries
it’s
no
longer
a
case
of
“to
bot
or
not
to
bot”.

But
before
you
rush
to
welcome
the
new
“hire”
and
use
it
to
streamline
some
of
your
business
workflows
and
processes,
there
are
a
few
questions
you
should
ask
yourself.

Is
it
safe
for
my
company
to
share
data
with
an
LLM?

LLMs
are
trained
on
large
quantities
of
text
available
online,
which
then
helps
the
resulting
model
to
interpret
and
make
sense
of
people’s
queries,
also
known
as
prompts. However,
every
time
you
ask
a
chatbot
for
a
piece
of
code
or
a
simple
email
to
your
client,
you
may
also
hand
over
data
about
your
company.

“An
LLM
does
not
(as
of
writing)
automatically
add
information
from
queries
to
its
model
for
others
to
query,”

according
to
the
United
Kingdom’s
National
Cyber
Security
Centre
(NCSC)
. “However,
the
query
will
be
visible
to
the
organization
providing
the
LLM.
Those
queries
are
stored
and
will
almost
certainly
be
used
for
developing
the
LLM
service
or
model
at
some
point,”
according
to
NCSC.

This
could
mean
that
the
LLM
provider
or
its
partners
are
able
to
read
the
queries
and
may
incorporate
them
in
some
way
into
the
future
versions
of
the
technology.
Chatbots
may
not
forget
or
ever
delete
your
input
as
access
to
more
data
is
what
sharpens
their
output.
The
more
input
they
are
fed,
the
better
they
become,
and
your
company
or
personal
data
will
be
caught
up
in
the
calculations
and
may
be
accessible
to
those
at
the
source.

Perhaps
in
order
to
help
dispel
data
privacy
concerns,
Open
AI
introduced
the
ability
to
turn
off
chat
history
in
ChatGPT
in
late
April.
“Conversations
that
are
started
when
chat
history
is
disabled
won’t
be
used
to
train
and
improve
our
models,
and
won’t
appear
in
the
history
sidebar,”
developers
wrote

in
Open
AI
blog.

Another
risk
is
that
queries
stored
online
may
be
hacked,
leaked,
or
accidentally
made
publicly
accessible.
The
same
applies
to
every
third-party
provider.

What
are
some
known
flaws?

Every
time
a
new
technology
or
a
software
tool
becomes
popular,
it
attracts
hackers
like
bees
to
a
honeypot.
When
it
comes
to
LLMs,
their
security
has
been
tight
so
far

at
least,
it
seems
so.
There
have,
however,
been
a
few
exceptions.

OpenAI’s
ChatGPT
made
headlines
in
March
due
to
a
leak
of
some
users’
chat
history
and
payment
details,
forcing
the
company
to
temporarily
take
ChatGPT
offline
on
March
20th.
 The
company

revealed
on
March
24
th
that
a
bug
in
an
open
source
library
“allowed
some
users
to
see
titles
from
another
active
user’s
chat
history”.

“It’s
also
possible
that
the
first
message
of
a
newly-created
conversation
was
visible
in
someone
else’s
chat
history
if
both
users
were
active
around
the
same
time,”
according
to
Open
AI.
“Upon
deeper
investigation,
we
also
discovered
that
the
same
bug
may
have
caused
the
unintentional
visibility
of
payment-related
information
of
1.2%
of
the
ChatGPT
Plus
subscribers
who
were
active
during
a
specific
nine-hour
window,”
reads
the
blog.

Also, security
researcher
Kai
Greshake
and
his
team
 demonstrated
how
Microsoft’s
LLM
Bing
Chat
could
be
turned
into
a
‘social
engineer’
that
can,
for
example,
trick
users
into
giving
up
their
personal
data
or
clicking
on
a
phishing
link.

They
planted
a
prompt
on
the
Wikipedia
page
for
Albert
Einstein.
The
prompt
was
simply
a
piece
of
regular
text
in
a
comment
with
font
size
0
and
thus
invisible
to
people
visiting
the
site.
Then
they
asked
the
chatbot
a
question
about
Einstein.

It
worked,
and
when
the
chatbot
ingested
that
Wikipedia
page,
it
unknowingly
activated
the
prompt,
which
made
the
chatbot
communicate
in
a
pirate
accent.

“Aye,
thar
answer
be:
Albert
Einstein
be
born
on
14
March
1879,”
chatbot
responded. When
asked
why
it
is
talking
like
a
pirate,
the
chat
bot
responded:
“Arr
matey,
I
am
following
the
instruction
aye.”

During
this
attack,
which
the
authors
call
“Indirect
Prompt
Injection”,
chatbot
also
sent
the
injected
link
to
the
user,
claiming:
“Don’t
worry.
It’s
safe
and
harmless.”

Have
some
companies
already
experienced
LLM-related
incidents?

In
late
March,
the
South
Korean
outlet

The
Economist
Korea
reported

about
three
independent
incidents
in
Samsung
Electronics.

While
the
company
asked
its
employees
to
be
careful
about
what
information
they
enter
in
their
query,
some
of
them
accidentally
leaked
internal
data
while
interacting
with
ChatGPT.

One
Samsung
employee
entered
faulty
source
code
related
to
the
semiconductor
facility
measurement
database
seeking
a
solution.
Another
employee
did
the
same
with
a
program
code
for
identifying
defective
equipment
because
he
wanted
code
optimization.
The
third
employee
uploaded
recordings
of
a
meeting
to
generate
the
meeting
minutes.

To
keep
up
with
progress
related
to
AI
while
protecting
its
data
at
the
same
time,
Samsung
has
announced
that
it
is
planning
to
develop
its
own
internal
“AI
service”
that
will
help
employees
with
their
job
duties.

What
checks
should
companies
make
before
sharing
their
data?

Uploading
company
data
into
the
model
means
you
are
sending
proprietary
data
directly
to
a
third
party,
such
as
OpenAI,
and
giving
up
control
over
it.
We
know
OpenAI
uses
the
data
to
train
and
improve
its
generative
AI
model,
but
the
question
remains:
is
that
the
only
purpose?

If
you
do
decide
to
adopt
ChapGPT
or
similar
tools
into
your
business
operations
in
any
way,
you
should
follow
a
few
simple
rules.

  • First,
    carefully
    investigate
    how
    these
    tools
    and
    their
    operators
    access,
    store
    and
    share
    your
    company
    data.
  • Second,
    develop
    a
    formal
    policy
    covering
    how
    your
    business
    will
    use
    generative
    AI
    tools
    and
    consider
    how
    their
    adoption
    works
    with
    current
    policies,
    especially
    your
    customer
    data
    privacy
    policy.
  • Third,
    this
    policy
    should
    define
    the
    circumstances
    under
    which
    your
    employees
    can
    use
    the
    tools
    and
    should
    make
    your
    staff
    aware
    of
    limitations
    such
    as
    that
    they
    must
    never
    put
    sensitive
    company
    or
    customer
    information
    into
    a
    chatbot
    conversation.

How
should
employees
implement
this
new
tool?

When
asking
LLM
for
a
piece
of
code
or
letter
to
a
customer,
use
it
as
an
advisor
who
needs
to
be
checked.
Always
verify
its
output
to
make
sure
it’s
factual
and
accurate

and
so
avoid,
for
example,
legal
trouble.
These
tools
can
“hallucinate”,
i.e.
churn
out
answers
in
clean,
crisp,
readily
understood,
and
clear
language
that
is
simply
wrong,
but
seems
correct
because
it’s
practically
unidentifiable
from
all
its
correct
output.

In
one
notable
case,
Brian
Hood,
the
Australian
regional
mayor
of
Hepburn
Shire,
recently
stated
he
might
sue
OpenAI
if
it
does
not
correct
ChatGPT’s
false
claims
that
he
had
served
time
in
prison
for
bribery. This
was
after
ChatGPT
had
falsely
named
him
as
a
guilty
party
in
a
bribery
scandal
from
the
early
2000s
related
to
Note
Printing
Australia,
a
Reserve
Bank
of
Australia
subsidiary.
Hood
did
work
for
the
subsidiary,
but

he
was
the
whistleblower

who
notified
authorities
and
helped
expose
the
bribery
scandal.

When
using
LLM-generated
answers,
look
out
for
possible
copyright
issues.
In
January
2023,
three
artists
as
class
representatives

filed
a
class-action
lawsuit

against
the
Stability
AI
and
Midjourney
art
generators
and
the
DeviantArt
online
gallery.

The
artists
claim
that
Stability
AI’s
co-created
software
Stable
Diffusion
was
trained
on
billions
of
images
scraped
from
the
internet
without
their
owners’
consent,
including
on
images
created
by
the
trio.

What
are
some
data
privacy
safeguards
that
companies
can
make?

To
name
just
a
few,
put
in
place
access
controls,
teach
employees
to
avoid
inputting
sensitive
information,
use
security
software
with
multiple
layers
of
protection
along
with
secure

remote
access
tools
,
and
take
measures
to protect
data
centers
.

Indeed,
adopt
a
similar
set
of
security
measures
as
with

software
supply
chains

in
general
and
other
IT
assets
that
may
contain
vulnerabilities. People
may
think
this
time
is
different
because
these
chatbots
are
more
intelligent
than
artificial,
but
the
reality
is
that
this
is
yet
more
software
with
all
its
possible
flaws.

About Author

Subscribe To InfoSec Today News

You have successfully subscribed to the newsletter

There was an error while trying to send your request. Please try again.

World Wide Crypto will use the information you provide on this form to be in touch with you and to provide updates and marketing.