Summary
Amazon
Web
Service
(AWS)
Lambda
is
a
serverless
event-driven
compute
service.
It
is
a
function
as
a
service
(FaaS)
that
allows
users
to
deploy
application
functionality
without
the
complexity
of
maintaining
the
underlying
infrastructure.
Lambda
executions
can
be
triggered
by
events
from
other
AWS
services
or
software-as-a-service
(SaaS)
applications.
Inside
the
Lambda
execution
environment
is
a
set
of
AWS
Security
Token
Service
(AWS
STS)
temporary
and
limited-privilege
credentials
for
AWS
Identity
and
Access
Management
(IAM).
An
attacker
may
be
able
to
steal
these
credentials
via
a
user
application
vulnerability
or
resource
misconfiguration.
The
attacker
could
then
use
these
credentials
to
escalate
privileges,
maintain
persistence,
or
move
laterally
through
an
organization’s
AWS
account
or
accounts.
Secureworks®
Counter
Threat
Unit™
(CTU)
researchers
have
developed
a
technique
using
AWS
CloudTrail
to
detect
the
use
of
stolen
credentials.
Every
time
an
AWS
Lambda
executes,
it
generates
an
AWS
CloudTrail
logging
event
that
can
be
used
to
establish
a
baseline
for
normal
operation.
The
use
of
stolen
credentials
can
then
be
detected
when
a
logging
event
deviates
from
the
baseline.
A
similar
approach
could
be
applied
to
detect
AWS
credentials
stolen
from
other
services.
Although
Amazon
GuardDuty
detects
EC2
instance
credentials
used
from
another
AWS
account,
it
does
not
apply
to
Lambda
or
any
other
services
nor
does
it
detect
credentials
being
used
within
the
same
account.
AWS
Lambda
execution
and
event
logging
Understanding
the
detection
logic
technique
requires
some
knowledge
of
the
AWS
Lambda
architecture,
operation,
and
management
API
calls.
AWS
CloudTrail
AWS
CloudTrail
monitors
and
records
account
activity
and
API
usage
across
all
AWS
accounts.
By
default,
CloudTrail
records
a
90-day
history
of
management
events
and
makes
this
data
freely
available
to
AWS
customers.
Customers
may
optionally
enable
data
event
logging
that
includes
AWS
Lambda
execution
and
Invoke
events,
but
this
feature
is
an
additional
cost.
Because
not
all
AWS
customers
may
enable
this
feature,
the
detections
need
to
rely
on
the
management
events
captured
in
all
environments.
Lambda
microVMs
and
Workers
According
to
AWS,
“Lambda
will
create
its
execution
environments
on
a
fleet
of
Amazon
EC2
instances
called
AWS
Lambda
Workers.
Workers
are
[…]
launched
and
managed
by
Lambda
in
a
separate
isolated
AWS
account
which
is
not
visible
to
customers.
Workers
have
one
or
more
hardware-virtualized
micro
virtual
machines
[(microVMs)]
created
by
Firecracker.”
Figure
1
shows
the
isolation
of
two
customers’
Lambda
functions
on
shared
infrastructure.
Figure
1.
Isolation
model
for
AWS
Lambda
Workers.
(Source:
AWS)
Lambda
execution
environment
lifecycle
The
standard
lifecycle
of
the
execution
environment
includes
three
primary
phases
(see
Figure
2).
Figure
2.
Lambda
execution
environment
lifecycle.
(Source:
AWS)
-
Init
—
Lambda
creates
or
unfreezes
an
execution
environment
with
the
configured
resources,
downloads
the
code
for
the
function
and
all
layers,
initializes
extensions,
initializes
the
runtime,
and
then
runs
the
function’s
initialization
code
(i.e.,
the
code
outside
the
main
handler).
The
Init
phase
happens
either
during
the
first
invocation
or
in
advance
of
function
invocations
if
provisioned
concurrency
is
enabled.
The
Init
phase
is
split
into
three
sub-phases:
Extension
init,
Runtime
init,
and
Function
init.
These
sub-phases
ensure
that
all
extensions
and
the
runtime
complete
their
setup
tasks
before
the
function
code
runs.When
a
Lambda
creates
an
execution
environment
during
the
Init
phase,
it
is
commonly
referred
to
as
the
‘cold
start.’
When
an
already
initialized
environment
is
invoked
again
before
shutdown
is
triggered,
it
is
called
a
‘warm
start.’ -
Invoke
—
Lambda
invokes
the
function
handler.
After
the
function
runs
to
completion,
Lambda
prepares
to
handle
another
function
invocation. -
Shutdown
—
This
phase
is
triggered
if
the
Lambda
function
does
not
receive
any
invocations
for
a
period
of
time.
It
is
unclear
what
logic
AWS
uses
to
calculate
this
timeframe.
In
the
Shutdown
phase,
Lambda
shuts
down
the
runtime,
alerts
the
extensions
to
let
them
stop
cleanly,
and
then
removes
the
environment.
According
to
AWS,
Workers
have
a
maximum
lease
time
of
14
hours.
However,
CTU
researchers
observed
much
shorter
lifecycles.
The
AWS
STS
credentials
used
by
the
Worker
have
a
default
expiration
of
12
hours,
so
it
is
probable
that
the
credentials
are
still
valid
after
a
Worker
is
shut
down.
AWS
introduced
Lambda
SnapStart
for
Java
11
runtime
in
November
2022.
Its
lifecycle
is
slightly
different
from
the
standard
Lambda
lifecycle.
However,
the
standard
lifecycle
is
suitable
for
this
analysis
because
CTU™
researchers
observed
no
differences
in
the
detection
logic.
Lambda
initialization
and
logging
Every
Lambda
cold
start
records
two
CloudTrail
events:
AssumeRole
and
CreateLogStream
(see
Figure
3).
Figure
3.
API
calls
during
Lambda
initialization.
(Source:
Secureworks)
-
The
AssumeRole
event
is
recorded
when
the
customer-defined
Lambda
identity
access
management
(IAM)
execution
role
requests
AWS
STS
credentials
from
the
invoke
service.
The
timing
of
this
event
in
the
execution
lifecycle
is
unclear,
but
CTU
research
shows
it
occurs
prior
to
initialization
of
the
Lambda
Worker. -
The
Lambda
Worker
makes
an
API
call
and
attempts
to
create
a
CloudWatch
log
stream.
If
the
CloudWatch
log
group
does
not
exist,
then
the
first
Lambda
execution’s
CreateLogStream
API
call
fails
with
error
code
ResourceNotFoundException.
The
Lambda
Worker
then
attempts
to
call
CreateLogGroup,
followed
by
a
successful
CreateLogStream
call.
The
detection
outlined
in
this
analysis
relies
on
the
following
assumptions:
-
The
Lambda
execution
IAM
role
is
configured
with
permissions
to
log
to
CloudWatch.
The
AWS
managed
policy
that
is
typically
used,
AWSLambdaBasicExecutionRole,
includes
these
permissions. -
If
the
CloudWatch
log
group
exists
but
the
Lambda
execution
role
does
not
have
logging
permission,
then
CloudTrail
will
still
log
the
failed
events. -
If
the
log
group
does
not
exist
and
the
execution
role
does
not
have
logging
permission,
then
CreateLogStream
generates
no
CloudTrail
events.
Because
the
Lambda
Worker
is
the
source
of
the
CreateLogStream
event,
the
event
includes
the
AWS
region
and
source
IP
address
where
the
customer’s
Lambda
function
executes.
The
source
of
the
AssumeRole
event
is
outside
the
Worker
and
therefore
does
not
contain
this
information.
Invoke
events
are
data
events
and
are
labeled
in
Figure
3
for
completeness,
but
they
are
excluded
from
the
following
detection
logic
because
they
may
not
exist
in
all
environments.
Figure
4
lists
an
example
AssumeRole
event
created
during
the
Lambda
Init
phase.
Figure
4.
AssumeRole
CloudTrail
event.
(Source:
Secureworks)
Figure
5
lists
an
example
CreateLogStream
event.
Figure
5.
CreateLogStream
CloudTrail
event.
(Source:
Secureworks)
Concurrency
Sometimes
multiple
CreateLogStream
events
possess
the
same
access
key
ID.
These
events
possibly
represent
the
Lambda
Worker
starting
multiple
processes
with
the
same
STS
credentials.
Figure
6
shows
the
diff
command
output
where
two
event
keys
match
but
the
IP
addresses
are
different.
Figure
6.
Output
of
diff
command
comparing
CreateLogStream
events.
(Source:
Secureworks)
Figure
7
is
an
example
of
a
Lambda
initialization
with
concurrency.
Figure
7.
API
calls
during
Lambda
initialization
with
concurrent
execution.
(Source:
Secureworks)
AWS
STS
key
reuse
CTU
researchers
discovered
that
AWS
STS
access
keys
can
be
reused
over
time
and
associated
with
unrelated
events.
The
two
events
in
Figure
8
possess
the
same
STS
access
key
ID
(accessKeyId),
but
the
credentials
were
assumed
by
different
user
identities
that
had
different
IAM
roles.
The
principalId
and
userName
fields
are
different
components
of
the
IAM
role’s
Amazon
resource
names
(ARNs).
The
time
difference
(‘creationDate’)
is
only
11
days.
It
is
possible
that
the
AWS
account
ID
is
used
as
input
to
the
access
key
generation
process.
Figure
8.
Output
of
diff
command
comparing
CloudTrail
events
for
different
roles
with
the
same
access
key
ID.
(Source:
Secureworks)
Amazon
Virtual
Private
Cloud
(VPC)
access
A
Lambda
function
runs
inside
a
VPC
owned
by
the
Lambda
service.
Lambdas
can
use
an
elastic
network
interface
(ENI)
to
connect
from
the
AWS-managed
Lambda
VPC
to
private
subnets
in
a
customer-managed
VPC.
When
a
Lambda
is
configured
to
access
resources
in
a
customer
VPC
and
security
groups
are
configured
to
allow
egress,
the
CreateLogStream
event’s
source
IP
address
will
not
match
the
public
source
IP
address
of
other
events.
Figure
9
is
an
example
of
a
complete
Lambda
configuration
with
VPC
access.
Each
number
corresponds
to
a
different
source
IP
address
in
the
CloudTrail
event
logs.
CreateNetworkInterface
and
AllocateAddress
events
are
generated
by
the
Lambda
Worker
when
connecting
to
the
customer
VPC.
Other
source
IP
addresses
may
appear
in
CloudTrail
when
a
Lambda
function
interacts
with
other
AWS
resources.
The
source
IP
address
for
a
request
to
AWS
S3
via
a
VPC
gateway
endpoint
is
the
private
IP
address
assigned
to
the
ENI.
The
request
to
DynamoDB
routes
via
the
internet,
and
the
source
is
the
public
IP
address
assigned
to
the
NAT
gateway.
Figure
9.
API
calls
during
Lambda
initialization
with
VPC
access.
(Source:
Secureworks)
The
VPC
access
scenario
uses
AllocateAddress
events
instead
of
CreateLogStream
to
find
the
public
IP
address,
specifically
the
responseElements.publicIp
field
(see
Figure
10).
Figure
10.
AllocateAddress
CloudTrail
event.
(Source:
Secureworks)
Proof-of-concept
detection
The
proof-of-concept
detection
uses
Amazon
Athena
to
identify
CloudTrail
events
associated
with
the
use
of
stolen
Lambda
credentials.
This
detection
is
portable
and
can
be
used
ad-hoc
in
any
AWS
account,
such
as
during
an
incident
response
investigation.
Detection
logic
The
first
step
in
building
the
detector
is
to
extract
metadata
from
the
CreateLogStream
events.
These
events
are
generated
by
the
Workers
during
the
Init
phase
of
the
Lambda
execution
lifecycle.
The
CloudTrail
logs
are
filtered
on
the
following
fields:
-
eventName
—
Filter
for
the
‘CreateLogStream’
value,
which
the
Worker
uses
to
set
up
CloudWatch
logging -
userAgent
—
Filter
for
strings
that
contain
the
‘awslambda-worker’
value
(e.g.,
awslambda-worker/1.0
rusoto/0.48.0
rust/1.67.1
linux)
A
table
is
created
using
these
events
to
form
a
baseline
of
standard
operation.
The
table
is
populated
with
the
following
metadata
fields:
-
eventTime
—
timestamp
of
the
CreateLogStream
event
(This
event
occurs
seconds
after
the
related
AssumeRole
event
and
can
be
used
to
infer
when
the
STS
credentials
were
generated.) -
userIdentity.accessKeyId
—
AWS
STS
access
key
ID
used
by
the
Worker
and
Lambda
function -
userIdentity.arn
—
ARN
of
the
IAM
role
associated
with
the
access
credentials,
used
to
deconflict
when
STS
access
key
IDs
are
reused -
sourceIPAddress
—
source
IP
address
of
the
event
(When
Lambda
concurrency
is
enabled,
multiple
IP
addresses
associated
with
Lambda
executions
use
the
same
credentials.) -
awsRegion
—
geographical
location
of
the
AWS
region
where
the
event
originated
(This
location
provides
additional
information
to
disambiguate
events.)
A
second
table
is
created
that
contains
all
customer-allocated
public
IP
addresses.
These
are
IP
addresses
that
have
been
associated
with
a
resource
in
an
AWS
account,
such
as
a
NAT
gateway
or
EC2
instance.
CloudTrail
logs
are
filtered
for
‘AllocateAddress’
events,
and
the
following
fields
are
extracted:
-
publicIp
—
IP
address
allocated
to
an
AWS
resource -
allocationId
—
unique
identifier
for
the
original
AllocateAddress
event -
·networkBorderGroup
—
location
where
the
IP
address
was
allocated
within
an
AWS
region
To
determine
if
credentials
have
been
exfiltrated
or
used
outside
the
standard
Lambda
execution,
a
query
is
run
against
all
CloudTrail
events.
It
looks
for
events
where
the
access
key
ID
exist
in
the
baseline
Lambda
events
table
and
where
the
following
criteria
is
met:
-
The
sourceIPAddress
is
different
from
the
IP
addresses
used
by
the
Lambda
Worker
in
the
Init
phase. -
The
userIdentity.arn
is
the
same
as
the
ARN
used
by
the
Lambda. -
The
eventName
is
not
Decrypt.
AWS
Key
Management
Service
(KMS)
generates
decrypt
events
if
enabled.
This
detector
ignores
these
events. -
The
eventTime
is
more
recent
than
the
Lambda
Init
phase. -
The
sourceIPAddress
has
not
been
observed
in
the
account.
False
positives
could
occur
if
sourceIPAddresses
are
very
old
and
are
not
within
the
log
retention
period.
This
condition
should
not
be
an
issue
with
this
detector
because
the
address
gets
allocated
to
the
ENI
at
approximately
the
same
time
as
the
CreateLogStream
event.
Applying
the
detection
using
Amazon
Athena
Before
AWS
CloudTrail
logs
can
be
queried
with
Amazon
Athena,
the
following
prerequisites
are
required:
-
Configure
a
trail
to
write
logs
to
an
S3
bucket
if
one
does
not
already
exist.
AWS
recommends
this
process
as
best
security
practice
to
store
and
retain
events
longer
than
90
days.
For
the
detection
to
be
accurate,
the
trail
should
be
configured
to
collect
logs
from
all
enabled
AWS
regions. -
Create
an
Athena
table
for
CloudTrail
logs.
The
default
table
name
is
‘cloudtrail_logs’.
If
a
different
table
name
is
used,
the
FROM
statements
in
the
Athena
queries
needs
to
be
updated
with
the
revised
name.
When
the
prerequisites
are
completed,
Amazon
Athena
can
search
CloudTrail
events
for
use
of
stolen
credentials.
The
Appendix
of
this
analysis
includes
these
Athena
queries
in
plain
text
for
researchers
who
want
to
replicate
the
detection.
-
Create
a
separate
table
containing
the
CreateLogStream
events
(see
Figure
11).
The
query
will
retrieve
the
earliest
eventTime
value
and
combine
the
source
IP
addresses
from
concurrent
execution
environments
into
an
array.
Figure
11.
Athena
query
to
create
table
for
CreateLogStream
events.
(Source:
Secureworks) -
Create
another
table
containing
all
of
the
public
IP
addresses
associated
with
customer
resources
(e.g.,
internet
gateway)
(see
Figure
12).
Figure
12.
Athena
query
to
create
table
for
customer-allocated
IP
addresses.
(Source:
Secureworks) -
Query
for
events
using
stolen
Lambda
credentials
(see
Figure
13).
Figure
13.
Athena
query
to
select
events
using
stolen
credentials.
(Source:
Secureworks)
Figure
14
shows
an
example
GetCallerIdentity
event
found
with
this
detection.
Figure
14.
Event
detected
using
stolen
credentials.
(Source:
Secureworks)
The
following
values
should
immediately
flag
this
event
as
suspicious:
-
awsRegion
“us-east-1”
does
not
match
“us-west-2” -
eventName
“GetCallerIdentity”
is
unlikely
to
be
run
by
legitimate
Lambda -
sourceIPAddress
“203
.
0
.
113
.
9”
does
not
match
“34
.
220
.
84
.
211” -
sourceIPAddress
“203
.
0
.
113
.
9”
is
not
within
the
AWS
IP
address
space
Taking
the
detector
further
AWS
publishes
its
public
IP
address
ranges
using
Classless
Inter-Domain
Routing
(CIDR)
notation
in
JSON
format.
This
data
could
increase
detection
logic
efficiency
by
first
checking
if
an
IP
address
exists
outside
AWS,
and
then
using
more
computationally
expensive
logic
to
check
other
conditions.
Similar
detection
logic
could
be
applied
to
other
services
within
AWS.
Primary
candidates
would
be
AWS
Elastic
Container
Service
(ECS)
or
Elastic
Kubernetes
Service
(EKS).
Both
services
can
leverage
the
AWS
Fargate
serverless
compute
engine,
which
is
built
on
the
same
Firecracker
microVM
as
the
Lambda
Worker.
Caveats
Lambda
pricing
is
calculated
per
one
million
requests.
Cold
starts
in
environments
with
a
high
frequency
of
executions
happen
less
often.
However,
real-time
detection
could
generate
a
large
quantity
of
detector
database
inserts
and
result
in
a
large
table
size.
The
detection
logic
described
in
this
analysis
relies
on
an
undocumented
feature.
At
any
time,
AWS
could
change
how
Lambda
executes.
Conclusion
AWS
CloudTrail
is
a
rich
source
of
management
events.
Network
defenders
can
use
specific
events
in
the
AWS
Lambda
operating
environment
to
provide
a
baseline
or
context
for
other
events.
The
proof-of-concept
detector
using
Athena
can
effectively
search
events
for
malicious
behavior.
Appendix
—
Athena
queries
Text
versions
of
the
Athena
queries
used
in
the
detection
proof
of
concept
are
provided
for
convenience
to
other
researchers
who
want
to
explore
this
functionality.
Create
table
for
CreateLogStream
events
CREATE TABLE "lambda_coldstart" AS
SELECT
useridentity.accessKeyId as accesskeyid,
-- Source IP can be different for the same access key id due to Lambda concurrency
array_agg(sourceipaddress) as sourceipaddresses,
awsregion,
useridentity.arn as arn,
array_agg(useragent) as useragents,
MIN(eventtime) as eventtime
FROM cloudtrail_logs WHERE
eventname = 'CreateLogStream' AND
useragent LIKE 'awslambda-worker%'
GROUP BY 1, 3, 4
Create
table
for
customer-allocated
IP
addresses
CREATE TABLE allocated_addresses AS
SELECT
json_extract_scalar(responseelements, '$.publicIp') as publicip,
json_extract_scalar(responseelements, '$.allocationId') as allocationid,
-- this is region name
json_extract_scalar(responseelements, '$.networkBorderGroup') as networkbordergroup
FROM
"cloudtrail_logs"
WHERE
eventname = 'AllocateAddress';
Select
events
using
stolen
credentials
SELECT
lcs.accesskeyid,
lcs.sourceipaddresses,
lcs.awsregion,
ct.useridentity.accessKeyId,
ct.sourceipaddress,
ct.awsregion,
ct.eventname,
ct.eventid
FROM cloudtrail_logs ct, lambda_coldstart lcs WHERE
lcs.accesskeyid = ct.useridentity.accessKeyId AND
not contains(lcs.sourceipaddresses, ct.sourceipaddress) AND
-- Exclude AWS managed services
ct.sourceipaddress != 'AWS Internal' AND
-- access keys can be reused to make sure it's the same ARN (which will differ)
ct.useridentity.arn = lcs.arn AND
-- Decrypt is noisy for the purposes of this detector
ct.eventname != 'Decrypt' AND
ct.eventtime > lcs.eventtime AND
-- Lookup IP addresses that have been allocated to account and exclude if they match
NOT EXISTS (SELECT 1 FROM allocated_addresses aa WHERE aa.publicip = ct.sourceipaddress);
About Author
