Error when parsing large memory dump

[BrianMer]

Hi,

I'm just trying to upload a (pretty large) raw memory dump file (almost 20 GB), but I have the following issue with the PostGres DB, which is causing the parsing not to finish, and the evidence being on "Unsatisfied requirements":

volweb-postgresdb  | 2025-01-03 15:24:37.878 UTC [4147] ERROR:  total size of jsonb array elements exceeds the maximum of 268435455 bytes at character 539
[...]
volweb-workers     | [2025-01-03 15:06:56,575: INFO/ForkPoolWorker-14] RUNNING: volatility3.plugins.windows.mftscan.MFTScan
volweb-workers     | [2025-01-03 15:06:56,605: INFO/ForkPoolWorker-14] Dependency for validation unavailable: jsonschema
volweb-workers     | [2025-01-03 15:24:43,166: ERROR/ForkPoolWorker-14] Task Windows.Engine[5635b07b-c9c8-4ed7-ad4a-e0889f7f4817] raised unexpected: UnboundLocalError("cannot access local variable 'e' where it is not associated with a value")
volweb-workers     | Traceback (most recent call last):
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
volweb-workers     |     return self.cursor.execute(sql, params)
volweb-workers     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
volweb-workers     | psycopg2.errors.ProgramLimitExceeded: total size of jsonb array elements exceeds the maximum of 268435455 bytes
volweb-workers     | LINE 1: ...volatility_engine_volatilityplugin"."artefacts" = '[{"__chil...
volweb-workers     |                                                              ^
volweb-workers     | 
volweb-workers     | 
volweb-workers     | The above exception was the direct cause of the following exception:
volweb-workers     | 
volweb-workers     | Traceback (most recent call last):
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/asgiref/sync.py", line 327, in main_wrap
volweb-workers     |     raise exc_info[1]
volweb-workers     |   File "/home/app/web/volatility_engine/engine.py", line 176, in start_extraction
volweb-workers     |     self.start_windows_analysis()
volweb-workers     |   File "/home/app/web/volatility_engine/engine.py", line 112, in start_windows_analysis
volweb-workers     |     self.run_plugin(builted_plugin)
volweb-workers     |   File "/home/app/web/volatility_engine/engine.py", line 82, in run_plugin
volweb-workers     |     constructed.run()
volweb-workers     |   File "/home/app/web/volatility_engine/plugins/windows/volweb_misc.py", line 66, in run
volweb-workers     |     self.run_all()
volweb-workers     |   File "/home/app/web/volatility_engine/plugins/windows/volweb_misc.py", line 60, in run_all
volweb-workers     |     renderer.render(self._grid)
volweb-workers     |   File "/home/app/web/volatility_engine/utils.py", line 313, in render
volweb-workers     |     self.save_to_database(final_output[1])
volweb-workers     |   File "/home/app/web/volatility_engine/utils.py", line 263, in save_to_database
volweb-workers     |     VolatilityPlugin.objects.update_or_create(
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method
volweb-workers     |     return getattr(self.get_queryset(), name)(*args, **kwargs)
volweb-workers     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/models/query.py", line 986, in update_or_create
volweb-workers     |     obj, created = self.select_for_update().get_or_create(
volweb-workers     |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/models/query.py", line 948, in get_or_create
volweb-workers     |     return self.get(**kwargs), False
volweb-workers     |            ^^^^^^^^^^^^^^^^^^
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/models/query.py", line 645, in get
volweb-workers     |     num = len(clone)
volweb-workers     |           ^^^^^^^^^^
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/models/query.py", line 382, in __len__
volweb-workers     |     self._fetch_all()
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/models/query.py", line 1928, in _fetch_all
volweb-workers     |     self._result_cache = list(self._iterable_class(self))
volweb-workers     |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/models/query.py", line 91, in __iter__
volweb-workers     |     results = compiler.execute_sql(
volweb-workers     |               ^^^^^^^^^^^^^^^^^^^^^
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1574, in execute_sql
volweb-workers     |     cursor.execute(sql, params)
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in execute
volweb-workers     |     return self._execute_with_wrappers(
volweb-workers     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
volweb-workers     |     return executor(sql, params, many, context)
volweb-workers     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/backends/utils.py", line 100, in _execute
volweb-workers     |     with self.db.wrap_database_errors:
volweb-workers     |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
volweb-workers     |     raise dj_exc_value.with_traceback(traceback) from exc_value
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
volweb-workers     |     return self.cursor.execute(sql, params)
volweb-workers     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
volweb-workers     | django.db.utils.OperationalError: total size of jsonb array elements exceeds the maximum of 268435455 bytes
volweb-workers     | LINE 1: ...volatility_engine_volatilityplugin"."artefacts" = '[{"__chil...
volweb-workers     |                                                              ^
volweb-workers     | 
volweb-workers     | 
volweb-workers     | During handling of the above exception, another exception occurred:
volweb-workers     | 
volweb-workers     | Traceback (most recent call last):
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/celery/app/trace.py", line 453, in trace_task
volweb-workers     |     R = retval = fun(*args, **kwargs)
volweb-workers     |                  ^^^^^^^^^^^^^^^^^^^^
volweb-workers     |   File "/home/app/.local/lib/python3.12/site-packages/celery/app/trace.py", line 736, in __protected_call__
volweb-workers     |     return self.run(*args, **kwargs)
volweb-workers     |            ^^^^^^^^^^^^^^^^^^^^^^^^^
volweb-workers     |   File "/home/app/web/volatility_engine/tasks.py", line 15, in start_extraction
volweb-workers     |     engine.start_extraction()
volweb-workers     |   File "/home/app/web/volatility_engine/engine.py", line 187, in start_extraction
volweb-workers     |     logger.warning(f"Unknown error, should not happen: {str(e)}")
volweb-workers     |                                                             ^
volweb-workers     | UnboundLocalError: cannot access local variable 'e' where it is not associated with a value

It seems like it is related with the MFTScan.

How can we bypass this issue?

[k1nd0ne]

Hello,

Hug... Seems that there is to many artefacts and that the database jsonfield size is exceeded + a typo in the exception handling.

I'll provide a fix as soon as possible.

Here is a quick fix for you:

Execute a shell inside the worker container image (docker exec -it CONTAINERID /bin/bash) and edit the backend/volatility_engine/volweb_misc.json file.

Delete this block of json:

"volatility3.plugins.windows.mftscan.MFTScan": {
        "icon": "Info",
        "description": "Retreive any MFT records residing in memory.",
        "category": "Filesystem",
        "display": "True"
      },

Re-upload your dump. You should be good to go.

I'll keep you posted when a fix is published (hopefully by the end of this week-end).

[k1nd0ne]

Hello there.

I have fixed the issue regarding the exception handling typo. However you are likely to face the problem again with the jsonfield size exceeded.

I am still looking for the best way to address this but no solution choosed at this moment.

Thank you again for this feedback!

I am keeping this opened for now.