How To Allowlist & Denylist Domains in a Rails app

This setup allows you to check incoming email submissions to any signup form. It uses a “trust & verify” approach— creating two separate tables: allowlisted domains and denylisted domains.

If a new user gives an email address that is on the allowlisted domain, they can continue with a certain set of activities automatically. These might include things like gmail.com, yahoo.com, etc.

If a new user gives an email address that is on neither the allowlisted domain nor the denylisted domains, they go into a limbo and have a limited set of things they can do — you determine what that is for these “greylisted” new signups. Typically, this means they have limited access until they confirm the email address using a confirmation link.

If a user gives an email that is on the denylisted list, on the other hand, they are stopped immediately. No confirmation email is sent, and the user record cannot not be created in the database.

For the greylisted domain names, which come in from the signup form, an automatic process runs every 6 hours to check to see if the domain name looks like it is valid. This process automatically allowlists if there is an A-record for the associated domain, but it does not denylist the domains — you must do that manually. Once you allowlist, the User’s domain_verified_at flag is set, and they can continue with the operations restricted to only domain verified users. If you denylist the domain, this code takes the aggressive strategy of deleting the user record. (If the domain was already on the denylist when the user signed up, they would not have have been able to signup).

This process is specifically about domain verification — just confirming that the domain supplied is valid to receive email messages at all. The only way to confirm that a full email is valid to actually send the confirmation email and require the user click on a confirmation link to confirm their account.

Domain verification is designed to stop you from sending bad emails to fraud domains — something that can and will hurt your email sending reputation. This process should be in addition to making the user actually confirm the email using a confirmation link. Once a domain is denylisted, no confirmation email will be sent to protect your email sending reputation (users will not be able to use that domain on the site).

1/ Add dnsruby Gem

bundle add dnsruby

2/ Add this patch to your String object

Create a file at lib/string_patch.rb

class String
  def obscure_email
    first_part = self.split("@")[0]
    first_part[0] + (first_part.length - 2).times.collect{"*"}.join + first_part[-1] + "@" + self.split("@")[1]
  end
end

require it in config/application.rb:

require "./lib/string_patch.rb"

3/ The DomainCheck Service Object

create app/services/domain_check.rb

class DomainCheck
  def self.check(domain)
    res = Dnsruby::Resolver.new

    begin
      ret = res.query(domain, Dnsruby::Types.A)
    rescue Dnsruby::NXDomain => e
      return false
    rescue Dnsruby::ResolvTimeout => e
      return false
    rescue Dnsruby::ServFail => e
      return false
    end

    return !!ret.answer[0]
  end
end

4/ Create Tables for Allowlisted and Denylisted Domains

rails generate model AllowListedDomain domain:string last_checked_at:datetime

rails generate model DenyListedDomain domain:string

class CreateAllowListedDomains < ActiveRecord::Migration[7.0]
  def change
    create_table :allow_listed_domains do |t|
      t.string :domain
      t.datetime :last_check_at
      t.timestamps
    end
  end
end

class CreateDenylistedDomains < ActiveRecord::Migration[7.0]
  def change
    create_table :denylisted_domains do |t|
      t.string :domain
      t.timestamps
    end
  end
end

5/ Add hooks to the AllowlistedDomain and DenylistedDomain model objects

app/models/allowlisted_domain.rb and

class AllowedDomain < ApplicationRecord
  before_destroy :reset_any_users

  def reset_any_users
    User.where("email ILIKE :search", search: "%@#{self.domain}" ).update_all(domain_verified_at: nil)
  end


  def to_label
    domain
  end
end

app/models/denylisted_domain.rb

class DenylistedDomain < ApplicationRecord
  after_create :kill_from_allowlist
  after_create :kill_any_users

  def kill_from_allowlist
   AllowlistedDomain.where(domain: self.domain).destroy_all
  end

  def kill_any_users
    User.where("email ILIKE :search", search: "%@#{self.domain}" ).destroy_all
  end

  def to_label
    domain
  end
end

6/ create file app/jobs/domain_verify_users_job.rb

require 'sidekiq-scheduler'
# require 'get_process_mem'
require 'dnsruby'


class DomainVerifyUsersJob < ApplicationJob
  queue_as :default
  def perform(*args)
    all_users = User.not_domain_verified
    count = all_users.count
    @email_result = ""
    if count > 0
      @email_result = "**** VERIFYING DOMAINS for #{count} unverified users.... "
    end
    all_users.each do |user|
      domain = user.email.split("@")[1]
      if !AllowListedDomain.find_by(domain: domain).nil?
        mark_allowlisted(user, domain)
      elsif !AllowListedDomain.find_by(domain: domain).nil?
        mark_allowlisted(user, domain)
      else
        if DomainCheck.check(domain)
          AllowListedDomain.create(domain: domain, last_checked_at: Time.current)
          @email_result << "\n Checking domain #{domain}... GOOD, adding to allowlist"
          mark_allowlisted(user, domain)
        else
          @email_result << "\n Checking domain #{domain}...\n BAD DOMAIN: \n denylisting #{domain} and deleting user #{user.id} #{user.email}"

          DenyListedDomain.create(domain: domain)
          user.destroy
        end
      end
    end
    if count > 0
      AdminMailer.with(result_data: @email_result, subject: "VDQ - DomainVerifyUsersJob").notice_email.deliver_now
    end
  end

  def mark_allowlisted(user, domain)
    @email_result << "\n Setting user #{user.email.obscure_email} to verified with domain #{domain}"
    user.domain_verified_at = Time.current
    user.save
  end

  def mark_denylisted(user, domain)
    @email_result << "\n Destroying user #{user.email} with denylisted domain #{domain}"
    user.destroy
  end
end

7/ Add domain_verified_at to your User model

class AddDomainVerifiedAtToUsers < ActiveRecord::Migration[7.0]
  def change
    add_column :users, :domain_verified_at, :datetime
  end
end

Add this scope to your user model:

scope :not_domain_verified, -> {where(domain_verified_at: nil)}

8/ Add logic to the controller that handles the signup when a user first enters an email address

Notice that if the domain is already denylisted, the user is not created at all.

If the domain is not denylisted, the user is created but the domain_verified_at will remain null.

If the domain is allowlisted, the user is created and the domain_verified_at will be set to the current time.

domain = user_params[:email].split("@")[1]
if DenylistedDomain.find_by(domain: domain)
  flash[:alert] = "Sorry, the domain #{domain} can't be used with this website. Please try a different email."
else

  allowlisted = ! AllowlistedDomain.find_by(domain: domain).nil?
  user = User.create!({email: user_params[:email],  domain_verified_at: (allowlisted ? DateTime.current : nil) })
  user.update(user_params)

  flash[:notice] = "Welcome to ____. Please check your email for confirmation instructions. "
  if !user.domain_verified_at
    flash[:notice] << "The domain @#{domain} has been flagged for review. Some emails may be blocked until this domain is verified. "
  end
end 

9/ You will need an Admin Mailer to send yourself an email

app/mailers/admin_mailer.rb

Note: instead of using Devise::Mailer as the super class you may also use a subclass of Devise::Mailer, such as AbcAppMailer. This will send you, the administrator, a daily report of the allowlisting & denylisting activity. Be sure to set your email using @@ADMIN_EMAIL below.

Note that because of how the job works (above), you will only get a message of any newly incoming domain was allowlisted or denylisted.

class AdminMailer < Devise::Mailer
  helper  :application # helpers defined within `application_helper`
  include Devise::Controllers::UrlHelpers # eg. `confirmation_url`

  default from: 'ADMIN <admin@domain.com>'
  default reply_to: 'admin@domain.com'
  layout  'admin_mailer'

  @@ADMIN_EMAIL = "your-admin-email@domain.com"
  def notice_email  # note the params are basically proxied to MailConstructor
    @content = params[:result_data]
    @subject = params[:subject]
    mail(to: @@ADMIN_EMAIL , subject: "#{@subject} [#{Rails.env}]")
  end
end

10/ Add To config/sidekiq.yml

Add this recurring job to the Sikekiq scheduler. This will run the DomainVerifyUsersJob every 6 hours.

:schedule:
  DomainVerifyUsersJob:
    cron: '1 */6 * * *'    # “At minute 1 past every 6th hour.”
    class: DomainVerifyUsersJob
    desc: "Domain Verify Users Job"

Use this sample denylisted domains file to get started:

You will need to load this CSV into the denylisted_domains table created above using a database tool.

11/ Build an Interface that Allows you to Manually Approve New Domains

To build an interface that allows you to go through the Users and allowlist or denylist the one that the job wouldn’t do, try out my gem Hot Glue.

Using Hot Glue, create admin interfaces to view & edit your AllowlistDomains and DenylistDomains, like so:

rails generate hot_glue:scaffold AllowlistDomain --gd --namespace=admin

rails generate hot_glue:scaffold DenylistDomain --gd --namespace=admin

The manual task for your admins to periodically is go through the Users where domain_verified_at is null and make a determination if the email domain looks valid. Add the domain name to either the allowlist or the denylist, then wait up to 6 hours for the job to run. Your user will get automatically domain verified or the user record will be destroyed accordingly.

Remember, this process should be in addition to making the user actually verify their email using a confirmation link. It protects you against obviously bad email inputs that can hurt your email sending score by letting you avoid sending to bad domains almost entirely or at little as possible.