Secure Model Primary Key ID
The Secure Model Primary Key ID package (securemodelpkid for short) provides mechanisms to help developers generate random primary keys for Django model objects. This modifies Django's default behaviour, which is to give each model an auto-incrementing, 64-bit integer primary key value starting from 1.
As object primary keys (henceforth referred to interchangeably as the object IDs) are often used directly in user-controlled input such as the URL path (e.g. /libraryapp/books/<id>), using randomly generated IDs that are difficult to enumerate is an effective way of defending against the exploitation of IDOR (Insecure Direct Object References) vulnerabilities.
Insecure Direct Object References (IDOR for short) is a vulnerability where user-controlled input is used directly to refer to objects within an application (e.g. records in a database). If the object identifier is easily guessable, an attacker could enumerate a list of valid identifiers and attempt to access the objects they refer to.
IDOR is usually not exploitable on its own, but can be dangerous if coupled with inadequate authorization or access control measures, potentially leading to unauthorized information disclosure that could result in further escalation. For more information on IDOR as well as concrete examples, refer to this article by PortSwigger.
Installation
The Secure Model Primary Key ID package can be installed using the command below:
pip install govtech-csg-xcg-securemodelpkid
Source code for the package can be found at https://github.com/GovTech-CSG/govtech-csg-xcg-securemodelpkid.
Usage
This package provides two methods for enabling randomly generated IDs:
- Making models inherit from a custom base class (requires slight code changes to models).
- Adding an app to the
INSTALLED_APPSsetting (requires no code changes to models).
Detailed instructions for each method are provided in the sections below.
If you are migrating an existing model class to use randomly generated IDs, please read the warning section at the end.
Method 1: Making models inherit from a custom base class
This method requires developers to modify their model classes to inherit from govtech_csg_xcg.securemodelpkid.model.RandomIDModel instead of the default Django base class (django.db.models.Model).
from django.db import models # models.Model is the usual base class, DON'T use this
from govtech_csg_xcg.securemodelpkid.model import RandomIDModel
# Our DB models will inherit from RandomIDModel instead of django.db.models.Model
class Customer(RandomIDModel):
name = models.CharField(max_length=50)
After modifying the model classes, run the shell commands below to ensure the changes are reflected in your database tables:
python manage.py makemigrations
python manage.py migrate
By default, this will generate a 32-character, URL-safe string as the object's primary key/ID, which is guaranteed to be unique. To modify the length, provide an integer value to the setting ID_TEXT_LENGTH in your settings.py file.
The minimum primary key length accepted by securemodelpkid is 18. Any value lower than 18 will be disregarded and the actual text length will be transparently set to 18.
Method 2: Adding an app to settings.INSTALLED_APPS
This method does not require any modifications to model classes. Instead, it requires the developer to add govtech_csg_xcg.securemodelpkid to the INSTALLED_APPS list in settings.py.
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
# ...
'govtech_csg_xcg.securemodelpkid',
]
This method will generate a 18-digit integer as the object's primary key/ID, which is guaranteed to be unique. Unlike method 1, there is no option to configure the length of the primary key/ID due to security considerations.
In order to use this method, the Django app containing your model must set the value of default_auto_field to django.db.models.BigAutoField. If you created your app using the python manage.py startapp command, this should already be the default case. See an example below:
from django.apps import AppConfig
class MyappConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'myapp'
Alternatively, if default_auto_field is not specified in the AppConfig object, Django will use the value of DEFAULT_AUTO_FIELD specified in the settings.py. If your project is created using the django-admin startproject command, this should be set as django.db.models.BigAutoField by default.
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
Performing data migrations
Django uses migrations to record and apply changes to your database. Migrations are usually automatically generated based on changes to your models when you run the command python manage.py makemigrations. Most of the time they represent changes to your database's schema, which is the structure of your tables and records. For example, a migration will be generated to change the column's data type if you modify your model's field from a CharField to an IntegerField.
A data migration is a special type of migration that modifies data in your database rather than changing its schema. Refer to the official Django documentation for more information on how to implement data migrations.
Data migration issue with RandomIDModel
Please read this section if one or more of your models inherit from RandomIDModel AND you need to perform a data migration on said models.
When performing a data migration on any model that inherits from RandomIDModel - specifically, when creating new objects - use the govtech_csg_xcg.securemodelpkid.helpers.generate_random_id function (see section below for the full function specification) to manually assign a random ID before calling the model's save method.
from django.db import migrations
from govtech_csg_xcg.securemodelpkid.helpers import generate_random_id
def create_record(apps, schema_editor):
"""Create some records as part of a data migration."""
# This is a model class that inherits from RandomIDModel.
CustomerWithRandomID = apps.get_model('app', 'CustomerWithRandomID')
CustomerWithRandomID(id=generate_random_id(CustomerWithRandomID), name="test").save()
class Migration(migrations.Migration):
dependencies = [
('app', '0001_initial'),
]
operations = [
migrations.RunPython(create_records)
]
If you fail to do the above, the resulting objects will be saved with a primary key value of '' (an empty string). If you are creating multiple objects in a loop, this will cause each subsequent object to overwrite the previous. For more information on why this happens, refer to the informational card below.
Django migrations record all changes applied to your databases, thereby functioning like a "version control system" for databases. The idea is that running all your migrations on a fresh database should restore it to your last recorded state. In order to achieve this, Django stores a historical version of your models for each point along the migration journey. Storage involves serialization, which is infeasible for arbitrary Python code, and as a result custom model methods are not stored.
from django.db import migrations
def create_record(apps, schema_editor):
"""Create some records as part of a data migration."""
# This is a model class that inherits from RandomIDModel.
CustomerWithRandomID = apps.get_model('app', 'CustomerWithRandomID')
CustomerWithRandomID(name="test").save()
class Migration(migrations.Migration):
dependencies = [
('app', '0001_initial'),
]
operations = [
migrations.RunPython(create_records)
]
When this migration is run, the RunPython operation will invoke the create_record function, which creates and saves a new CustomerWithRandomID object. Since CustomerWithRandomID inherits from RandomIDModel, we should expect the record to be created with a random string as its primary key value. However, the actual record is created with a primary key value of '' (an empty string), because the custom save method implemented by RandomIDModel is unavailable within the historical model when running code via RunPython.
The generate_random_id helper function
The generate_random_id helper function exposes the random ID generation functionality used by both RandomIDModel and the govtech_csg_xcg.securemodelpkid Django app. It can be found in the module govtech_csg_xcg.securemodelpkid.helpers. The function definition is as follows:
def generate_random_id(model, type='str'):
"""Generates a collison-free random ID for a model class.
Args:
model: A model class that inherits from django.db.models.Model.
This includes classes that inherit from RandomIDModel as
RandomIDModel also inherits from the default Django model class.
type: Either 'str', which generates a random string (used by RandomIDModel),
or 'int', which generates a random 64-bit integer (used by the
govtech_csg_xcg.securemodelpkid app). The generated values conform
to the lengths as defined in 'method 1' and 'method 2' above. The
default value is 'str'.
Returns:
Either a randomly generated string or a randomly generated 64-bit integer,
depending on the value of 'type'.
"""
You should not have to use this function under normal circumstances. Its main use case is to address data migration issues when using RandomIDModel. See the section above for more information.
Modifying existing models to use random IDs
This section serves as a warning for users who want to update the primary key values of existing objects to use randomly-generated IDs from this package.
Using securemodelpkid when creating new model classes is straightforward. By adopting either of the 2 methods above, any new objects created from the new model classes will have randomly generated primary key values. The issue arises when modifying an existing model class with existing objects (i.e. database records) to use securemodelpkid; While newly created objects will have random primary key values, any existing objects will not have their primary keys automatically updated to be consistent with the modification.
Unfortunately, Django does not provide a simple mechanism for updating primary keys. Attempting to "update" the primary key value by instantiating the object, modifying its primary key property, and then invoking the save method will create a copy of the original object with the randomly-generated primary key value instead of replacing it. For example:
from myapp.models import MyModel
from govtech_csg_xcg.securemodelpkid.helpers import generate_random_id
instance = MyModel.objects.get(id=1)
instance.id = generate_random_id(MyModel, type='str')
instance.save() # Creates a NEW record with the randomly generated ID
# The OLD record with id=1 still exists in the database
To resolve this, you could delete the object with the original primary key value. However, doing so runs the risk of severing relationships with other model classes that use your class' primary key as a foreign key - i.e. any model class with django.db.models.ForeignKey, django.db.models.OneToOneField, or django.db.models.ManyToManyField referring to the target class. If any of these relationships specify ON DELETE CASCADE semantics, deleting an object could even lead to unexpected deletion of related objects belonging to other models.
Due to the difficulties involved, we would recommend not using this package for existing models whose primary key is referenced by other models. However, if you would like to go ahead anyway, below are the high level steps required to update primary key values for a specific model in a safe manner.
- Back up your database so that you can roll back if anything goes wrong.
- Take note of all other model classes that have a
ForeignKey,OneToOneField, orManyToManyFieldfield referencing the target model. - Write a script that iterates through each existing object belonging to the target model, performing the actions below:
- Make a deep copy of the object
- "Modify" the copy's primary key value via the instance property (i.e.
object.pk = randomly_generated_value) and invoke thesavemethod - this creates a new record in the database while the original continues to exist - Obtain the original object's related objects (either using related managers or by directly referencing object property) and update their references to point to the copy
An example implementation of this is shown below:
"""
An example model set up, with the target model being the "Blog" class,
and other models with various types of relationships with it.
"""
from django.db import models
from govtech_csg_xcg.securemodelpkid.model import RandomIDModel
class Blog(RandomIDModel):
name = models.CharField(max_length=50)
"""
Many-to-one relationship
Each blog can have multiple entries
Each entry can only belong to one blog
"""
class Entry(models.Model):
title = models.CharField(max_length=50)
blog = models.ForeignKey(Blog, on_delete=models.CASCADE)
"""
Many-to-many relationship
Each blog can fall under multiple categories
Each category can house multiple blogs
"""
Many-to-many relationship
class Category(models.Model):
name = models.CharField(max_length=50)
blogs = models.ManyToManyField(Blog)
"""
One-to-one relationship
Each blog can optionally be an AdventureBlog, or some other kind of blog
Each AdventureBlog is a type of a Blog
Note: This case is equivalent to direct inheritance (i.e. class AdventureBlog(Blog))
When you set up a direct inheritance, Django implicitly creates a OneToOneField relationship
"""
class AdventureBlog(models.Model):
name = models.CharField(max_length=50)
blog = models.OneToOneField(Blog, on_delete=models.CASCADE, primary_key=True)
from copy import deepcopy
from govtech_csg_xcg.securemodelpkid.helpers import generate_random_id
from testapp.models import Blog
def update_existing_pks():
"""Update the primary key values for existing instances of the Blog model."""
for blog in Blog.objects.all():
# Create a new blog instance as a copy of the original.
updated_blog = deepcopy(blog)
updated_blog.pk = generate_random_id(Blog, type='str')
updated_blog.save()
# Modify the foreign key relationship for the Entry instances.
# The `set` operation uses an update SQL statement to modify foreign key values on all related "entries"
updated_blog.entry_set.set(blog.entry_set.all())
# Modify the many-to-many relationships for the Category instances.
updated_blog.category_set.set(blog.category_set.all())
blog.category_set.clear()
# Modify the one-to-one relationships for the AdventureBlog instances.
if hasattr(blog, 'adventureblog'):
updated_blog.adventureblog = blog.adventureblog
"""
This automatically creates a new AdventureBlog instance
with the same primary key our updated Blog.
If there are other models that refer to AdventureBlog,
we would have to follow those links and perform this
entire data migration for AdventureBlog first.
"""
updated_blog.adventureblog.save()
# Finally, delete the old instance
blog.delete()
update_existing_pks()
In the example above, you would run the migration script after the schema migration for changing to RandomIDModel has been completed. i.e. The sequence of events should be as follows:
- Change the model code for
Blogto inherit fromRandomIDModelinstead ofdjango.db.models.Model - Create and run a migration to update the database schema:
python manage.py makemigrations myapp && python manage.py migrate
- Run the helper script to update primary key values for all existing
Blogobjects:
python manage.py shell < migration_script.py
The example helper script above is not foolproof. Due to the complexities of Django's ORM system, there could be edge cases not considered by this reference implementation. Make sure to back up your data and perform proper testing before running any actions against your production database.