Skip to main content

Secure File Upload

The Secure File Upload package provides a Django middleware that validates files uploaded by users through forms and analyses them for signs of maliciousness. It performs the validation and analysis through a series of checks, including but not limited to:

  • Checking if the file type is whitelisted.
  • Checking if the file size is within limits.
  • Checking if the file contains potentially malicious content.

If a file is detected as invalid or malicious, the upload can be blocked within a view by checking an attribute of the Django request object.

Installation​

The Secure File Upload package can be installed using the command below:

pip install govtech-csg-xcg-securefileupload

Source code for the package can be found at https://github.com/GovTech-CSG/govtech-csg-xcg-securefileupload.

Extra dependencies

Installing the package govtech-csg-xcg-securefileupload will not install the yara-python and quicksand dependencies by default, due to some potential installation difficulties for those packages. However, yara-python is required if you want to use the YARA functionality of securefileupload, and quicksand is required for the Quicksand file analysis functionality. To install securefileupload with either optional dependencies, use the format shown below, either directly on the command line with pip install or in the requirements.txt file:

govtech-csg-xcg-securefileupload[yara]

or...

govtech-csg-xcg-securefileupload[quicksand]

or...

govtech-csg-xcg-securefileupload[yara,quicksand]

Setup​

To activate the middleware, add it to your MIDDLEWARE list in settings.py.

./<PROJECT_NAME>/settings.py
MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    # ...
    'govtech_csg_xcg.securefileupload.middleware.FileUploadValidationMiddleware'
]

Logging​

The package uses its own logger with the name securefileupload. By default, this logger uses a stream handler and emits messages with level INFO and above. The log message format looks something like this:

2023-06-22 09:50:43,348 [INFO][XCG][securefileupload]: This is an informational message.

To safely override the default logging configuration, you can write code in settings.py to import the logger from the package and modify it directly. The example below sets the logger's level to DEBUG:

settings.py
from govtech_csg_xcg.securefileupload import logger
import logging

logger.setLevel(logging.DEBUG)

For more information on the Python logging library, see the official documentation.

Usage​

Basics​

Once the middleware has been activated, it will analyse and validate any file that a user tries to upload via a form. If the validation fails, two special attributes will be set on the Django request object that is passed to your view

  1. block_upload: This will be set to True if the validation fails
  2. upload_errmsg: This will be set to a string containing the reason for failure

The following code snippet can then be used in your view to check for failures and respond appropriately.

./<APP_NAME>/views.py
def view_that_processes_file_uploads(request):
    if getattr(request, 'block_upload', False):
        err_msg = request.upload_errmsg
        respond_to_failure(err_msg)

Using xcg_file_validator​

If you use Django forms that are tied to models (i.e. form classes that inherit from django.forms.ModelForm), you can use the validator xcg_file_validator in your model class to avoid having to manually check the request.block_upload attribute in your views. In the example below, we have a form that creates/updates an "article" when submitted, which in turn requires a "cover photo":

./<APP_NAME>/forms.py
from django.forms import ModelForm
from .models import Article

class ArticleForm(ModelForm):
    class Meta:
        model = Article
        fields = ["pub_date", "headline", "cover_photo"]
./<APP_NAME>/models.py
from django.db import models
from govtech_csg_xcg.securefileupload.validators import xcg_file_validator

class Article(models.Model):
    pub_date = models.DateTimeField(default=timezone.now)
    headline = models.TextField(max_length=50)
    cover_photo = models.ImageField(upload_to="uploads/", validator=[xcg_file_validator])

If the image uploaded fails the validation, a call to the standard Django form validation method form.is_valid() will return False, which means the view code does not have to explicitly check if request.block_upload is True. And the upload_errmsg will be added to the forms error messages.

./<APP_NAME>/views.py
def submit_article(request):
    if request.method == 'POST':
        form = ArticleForm(request.POST, request.FILES)
        if form.is_valid():
            form.save()
            return redirect('/')
    else:
        form = ArticleForm()

    return render(request, 'article_submission.html', context={'form': form})
What if I forget to check for blocked uploads?

By default, the Secure File Upload middleware will replace files deemed to be malicious with a single-byte dummy file, so that the file upload process can run to completion. This means that even if you forget to check for the block_upload flag, or forget to use the xcg_file_validator, your app should still be safe from malicious uploads.

That said, we highly encourage using either one of the methods above to explicitly handle the failed upload, rather than letting the upload fail silently.

Configuration​

Available options​

Secure File Upload provides a number of configuration options that allow you to customise which checks are active. The default configuration will be used if no options are explicitly provided.

XCG_SECUREFILEUPLOAD_CONFIG = {
  "quicksand": False,
  "file_size_limit": None,
  "filename_length_limit": None,
  "whitelist_name": "RESTRICTIVE",
  "whitelist": [],
  "sanitization": True,
  "keep_original_filename": False,
  "clamav": False,
  "yara_file_location": ... # will point to builtin yara rules
}

Below are explanations for each of the options shown above:

  • quicksand: A Python-based analysis framework to analyse and identify exploits in suspected malware documents. Supports documents, PDFs, Mime/Email, Postscript and other common formats. Refer to the official documentation for more details.
  • file_size_limit: An integer that defines the maximum allowed file size in kilobytes. Files larger than this limit will be rejected.
  • filename_length_limit: An integer that defines the maximum allowed character length of the file name.
  • whitelist_name: Either "CUSTOM" or the name of a predefined whitelist that Secure File Upload provides. If using a predefined whitelist, the developer does not have to provide any values for the whitelist key. The predefined whitelists available are:
    • "ALL" - All the permissive whitelists below combined:
      • "AUDIO_ALL" - All audio files
      • "APPLICATION_ALL" - All application files
      • "IMAGE_ALL" - All image files
      • "TEXT_ALL: All text files
      • "VIDEO_ALL" - All video files
      • "ALL" - All files
    • "RESTRICTIVE" - All the restrictive whitelists below combined:
      • "AUDIO_RESTRICTIVE" - audio/mpeg
      • "APPLICATION_RESTRICTIVE" - application/pdf
      • "IMAGE_RESTRICTIVE" - image/gif, image/jpeg, image/png, image/tiff
      • "TEXT_RESTRICTIVE" - text/plain
      • "VIDEO_RESTRICTIVE" - video/mp4, video/mpeg
  • whitelist: Only required if the whitelist_name provided is "CUSTOM". This should be a list of MIME types (e.g. ['image/png', 'application/pdf']).
  • sanitization: Either True or False. If set to True, the middleware will sanitize the uploaded file automatically. e.g. rename it as a random UUID, sanitize PDF keywards (e.g. disabling of JavaScript execution), etc.
  • keep_original_filename: Either True or False. If set to True, files will be saved with their original name even if sanitization is set to True. Note that if a file with the same name exists, Django will rename it by adding a random suffix.
  • clamav: Either True or False. If set to True, the middleware will attempt to scan the file using ClamAV, providing that the ClamAV daemon is installed. See the section "clamd Install" in this document for installation instructions.
  • yara_file_location: By default, the middleware will use the builtin YARA rules stored within the package directory. Developers can provide customized files by specifying the absolute path to the folder that contain the YARA files. A collection of useful YARA signatures can be found in the awesome-yara repository

Global configuration​

A global configuration can be defined using the setting XCG_SECUREFILEUPLOAD_CONFIG in settings.py.

./<PROJECT_NAME>/settings.py
XCG_SECUREFILEUPLOAD_CONFIG = {
  "quicksand": True,
  "whitelist_name": "CUSTOM",
  "whitelist": ['application/pdf'],
  "clamav": False,
  "yara_file_location": BASE_DIR / 'yara'
}

Note that not all of the configuration options must be defined. Rather, you can define just a subset of the options for which you would like to override the default values.

View-specific configuration​

Each view can also define their own specific file upload configuration by using the decorator upload_config.

./<APP_NAME>/views.py
from govtech_csg_xcg.securefileupload.decorator import upload_config

@upload_config(quicksand=True, whitelist_name="CUSTOM", whitelist=['image/png'])
def upload_image(request):
    # ...

The supported configurations are quicksand, file_size_limit, filename_length_limit, whitelist_name and whitelist.

As with the global configuration, not all options need to be specified when using the decorator. An additional point to note is that the view-specific configuration takes precedence over the global configuration, thus allowing developers to specify granular validation criteria for each view that deals with file uploads.