Map DynamoDB Items to Objects using DynamoDB mapper.

Previously we created DynamoDB Tables using Java.

For various databases such sql databases or nosql there is a set of tools that help to access, persist, and manage data between objects/classes and the underlying database. For example for SQL databases we use JPA, for Cassandra we use MappingManager.

DynamoDBMapper is a tool that enables you to access your data in various tables, perform various CRUD operations on items, and execute queries and scans against tables.

We will try to map the Users, Logins, Supervisors and companies tables from the previous example.
Users is a simple table using the users email as a Hash key.

package com.gkatzioura.dynamodb.mapper.entities;

import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBAttribute;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBHashKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTable;

/**
 * Created by gkatzioura on 9/20/16.
 */
@DynamoDBTable(tableName="Users")
public class User {

    private String email;
    private String fullName;

    @DynamoDBHashKey(attributeName="email")
    public String getEmail() {
        return email;
    }

    @DynamoDBAttribute(attributeName="fullname")
    public void setEmail(String email) {
        this.email = email;
    }

    public String getFullName() {
        return fullName;
    }

    public void setFullName(String fullName) {
        this.fullName = fullName;
    }
}

However in various cases our DynamoDB Table uses a hash and a range key. The Logins table keeps track of the login attempts of a user. The email is the hash key and the timestamp the range key.

package com.gkatzioura.dynamodb.mapper.entities;

import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBHashKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBRangeKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTable;

/**
 * Created by gkatzioura on 9/20/16.
 */
@DynamoDBTable(tableName="Logins")
public class Login {

    private String email;
    private Long timestamp;

    @DynamoDBHashKey(attributeName="email")
    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }

    @DynamoDBRangeKey(attributeName="timestamp")
    public Long getTimestamp() {
        return timestamp;
    }

    public void setTimestamp(Long timestamp) {
        this.timestamp = timestamp;
    }
}

Another popular case are tables with global secondary indexes (GSI). For example the Supervisors table is used to retrieve a supervisor by his name. However we also use this table in order to retrieve all the supervisors from a specific company or the supervisors who work on specific factory of a company.
The supervisor name is our hash key, the company name is the hash key and the factory name is the range key of the global secondary index.

package com.gkatzioura.dynamodb.mapper.entities;

import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBHashKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBIndexHashKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBIndexRangeKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTable;

/**
 * Created by gkatzioura on 9/21/16.
 */
@DynamoDBTable(tableName="Supervisors")
public class Supervisor {

    private String name;
    private String company;
    private String factory;

    @DynamoDBHashKey(attributeName="name")
    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    @DynamoDBIndexHashKey(globalSecondaryIndexName = "FactoryIndex",attributeName = "company")
    public String getCompany() {
        return company;
    }

    public void setCompany(String company) {
        this.company = company;
    }

    @DynamoDBIndexRangeKey(globalSecondaryIndexName = "FactoryIndex",attributeName = "factory")
    public String getFactory() {
        return factory;
    }

    public void setFactory(String factory) {
        this.factory = factory;
    }
}

Last but not least we can use Local Secondary Indexes. The Companies table uses the company name as a hash key and the subsidiary name as a range key. Since we want to issue queries based on a company’s CEOs a local secondary index is used with a range key based on the name of the CEO.

package com.gkatzioura.dynamodb.mapper.entities;

import com.amazonaws.services.dynamodbv2.datamodeling.*;

/**
 * Created by gkatzioura on 9/21/16.
 */
@DynamoDBTable(tableName="Companies")
public class Company {

    private String name;
    private String subsidiary;
    private String ceo;

    @DynamoDBHashKey(attributeName="name")
    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    @DynamoDBRangeKey(attributeName = "subsidiary")
    public String getSubsidiary() {
        return subsidiary;
    }

    public void setSubsidiary(String subsidiary) {
        this.subsidiary = subsidiary;
    }

    @DynamoDBIndexRangeKey(localSecondaryIndexName = "CeoIndex",attributeName = "ceo")
    public String getCeo() {
        return ceo;
    }

    public void setCeo(String ceo) {
        this.ceo = ceo;
    }
}

You can find the source code on github.

Spring boot with Spring Security and jdbc Part 2

On a previous post we implemented security based on the default table schemas that Spring Security issues requests.

Considering users and roles, application developers use a schema that fits their needs. Spring gives us the ability to specify the queries needed in order to retrieve information such as username, password and roles.

Our custom tables will be pretty different from the tables of the first example.

drop table if exists Custom_Users;
create table Custom_Users(id bigint auto_increment, username varchar(255), password varchar(255));
insert into Custom_Users(username,password) values('TestUser','TestPass');

drop table if exists Custom_Roles;
create table Custom_Roles(username varchar(255),authority  varchar(255), UNIQUE(username,authority));
insert into Custom_Roles(username,authority) values('TestUser','superadmin');

In order to use these tables with spring security we must pass the queries that spring security will use in order to retrieve the security information needed.

To do so we will create a security configuration that will set up the queries needed.

package com.gkatzioura.spring.security.config;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Profile;
import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;

import javax.sql.DataSource;


/**
 * Created by gkatzioura on 9/20/16.
 */
@EnableWebSecurity
@Profile("customquery")
public class CustomQuerySecurityConfig extends WebSecurityConfigurerAdapter {

    @Autowired
    private DataSource dataSource;

    @Autowired
    public void configAuthentication(AuthenticationManagerBuilder auth) throws Exception {
        auth.jdbcAuthentication().dataSource(dataSource)
                .usersByUsernameQuery("SELECT username,password,1 FROM Custom_Users where username=?")
                .authoritiesByUsernameQuery("SELECT username,authority FROM Custom_Roles where username=?");
    }

    @Override
    protected void configure(HttpSecurity http) throws Exception {

        http.authorizeRequests()
                .antMatchers("/public").permitAll()
                .anyRequest().authenticated()
                .and()
                .formLogin()
                .permitAll()
                .and()
                .logout()
                .permitAll();
    }

}

We use spring profiles. Our spring profile would be “customquery”, therefore the CustomQuerySecurityConfig would be bound to the “customquery” profile.

In order to run, for convenience reasons we have to change the default profile in our build.gradle file.

group 'com.gkatzioura'
version '1.0-SNAPSHOT'

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:1.4.0.RELEASE")
    }
}

apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'spring-boot'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    compile("org.springframework.boot:spring-boot-starter-web")
    compile("org.thymeleaf:thymeleaf-spring4")
    compile("org.springframework.boot:spring-boot-starter-security")
    compile("org.springframework:spring-jdbc")
    compile("com.h2database:h2:1.4.192")
    compile("org.slf4j:slf4j-api:1.6.6")
    compile("ch.qos.logback:logback-core:1.1.7")
    compile("ch.qos.logback:logback-classic:1.1.7")
    testCompile "junit:junit:4.11"
}

bootRun {
    systemProperty "spring.profiles.active", "customquery"
}

To run the application issue

gradle bootRun

You can find the source code on github

Spring boot with Spring Security and jdbc

Spring security Is a wonderful framework saving lots of time and effort from the developers. Also It is flexible enough to customize and bring it down to your needs.

Working with JDBC and Spring Security is pretty easy and many actions are automated. This would be a minimal showcase.

The gradle file contains dependencies such as spring-security, spring-jdbc and h2 database. Since there would be a series of articles bootRun will set a spring profile on startup, In our case the profile called simple.

group 'com.gkatzioura'
version '1.0-SNAPSHOT'

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:1.4.0.RELEASE")
    }
}

apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'spring-boot'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    compile("org.springframework.boot:spring-boot-starter-web")
    compile("org.thymeleaf:thymeleaf-spring4")
    compile("org.springframework.boot:spring-boot-starter-security")
    compile("org.springframework:spring-jdbc")
    compile("com.h2database:h2:1.4.192")
    compile("org.slf4j:slf4j-api:1.6.6")
    compile("ch.qos.logback:logback-core:1.1.7")
    compile("ch.qos.logback:logback-classic:1.1.7")
    testCompile "junit:junit:4.11"
}

bootRun {
    systemProperty "spring.profiles.active", "simple"
}

Tables containing certain information must be created. Those tables will have the default name and column names that Spring security lookups in order to get information.

drop table if exists users;
create table users(id bigint auto_increment, username varchar(255), password varchar(255), enabled boolean);
insert into users(username,password,enabled) values('steve','steve',true);
insert into users(username,password,enabled) values('john','john',true);
drop table if exists authorities;
create table authorities(username  varchar(255),authority  varchar(255), UNIQUE(username,authority));
insert into authorities(username,authority) values('steve','admin');
insert into authorities(username,authority) values('john','superadmin');

Those sql statements will reside on resources/schema.sql.

First step is to create our Application class

package com.gkatzioura.spring.security;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

/**
 * Created by gkatzioura on 9/2/16.
 */
@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

}

In order to get started quickly the database will be an h2 database.

package com.gkatzioura.spring.security.config;

import org.h2.jdbcx.JdbcDataSource;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.sql.DataSource;
import java.nio.file.Files;

/**
 * Created by gkatzioura on 9/2/16.
 */
@Configuration
public class DataSourceConfig {

    @Bean
    public DataSource createDataSource() {

        JdbcDataSource dataSource = new JdbcDataSource();
        dataSource.setURL("jdbc:h2:"+System.getProperty("java.io.tmpdir")+"/database");

        return dataSource;
    }

}

By specifying the h2 database I set the directory to be inside the temporary directory. Therefore once you restart your os the database will be gone.
As mentioned previously once the datasource bean has been initialized spring-jdbc will automatically lookup on the resource folder for a schema.sql file. In case the file exists spring-jdbc will try to execute the statements that the schema.sql contains.

Next step is to define our security configuration. We have to specify that our security will be based on jdbc. Also we must define the endpoints that will have to be secure. This security config bean will be bound to the simple profile.

package com.gkatzioura.spring.security.config;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;

import javax.sql.DataSource;

/**
 * Created by gkatzioura on 9/2/16.
 */
@EnableWebSecurity
@Profile("simple")
public class SimpleSecurityConfig extends WebSecurityConfigurerAdapter {

    @Autowired
    private DataSource dataSource;

    @Autowired
    public void configAuthentication(AuthenticationManagerBuilder auth) throws Exception {
        auth.jdbcAuthentication().dataSource(dataSource);
    }

    @Override
    protected void configure(HttpSecurity http) throws Exception {

        http.authorizeRequests()
                .antMatchers("/public").permitAll()
                .anyRequest().authenticated()
                .and()
                .formLogin()
                .permitAll()
                .and()
                .logout()
                .permitAll();
    }
}

Last but not least we will add a controller with a secured endpoint and a non-secured endpoint

package com.gkatzioura.spring.security.controller;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

/**
 * Created by gkatzioura on 9/2/16.
 */
@RestController
public class GreetController {

    private static final Logger LOGGER = LoggerFactory.getLogger(GreetController.class);

    @RequestMapping(path = "/public",method = RequestMethod.GET)
    public String sayFreeHi() {
        return "Greeting";
    }

    @RequestMapping(path = "/secured",method = RequestMethod.GET)
    public String saySecureHi() {
        return "Secured";
    }

}

Once you try to access the secured endpoint the default spring security login screen will be displayed.
Proceed with one of the users specified in the sql statements (for example username: steve password: steve). In case you want to logout just hit the /login?logout endpoint.

Run the application with a

gradle bootRun

and you are good to go.

You can find the source code on github

Update DynamoDB Items with Node.js

On a previous post we proceeded into inserting items to DynamoDB using Node.js. DynamoDB also supports updating items.

We will use the Login table for the update examples.
When issuing an update you must specify the primary key of the item you want to update.

var updateName = function(email,fullName,callback) {
	
	var docClient = new AWS.DynamoDB.DocumentClient();
	
	var params = {
			TableName:"Users",
			Key: {
				email : email
			},
			UpdateExpression: "set fullname = :fullname",
		    ExpressionAttributeValues:{
		        ":fullname":fullName
		    },
		    ReturnValues:"UPDATED_NEW"
		};
	
	docClient.update(params,callback);
}

We can proceed on more advanced statements using conditional updates. Conditional updates can help us in many cases such as handling concurrent updates. In our case we will update an item’s Full name only if it starts with a certain prefix.

var updateConditionally = function(email,fullName,prefix,callback) {
	
	var docClient = new AWS.DynamoDB.DocumentClient();
	
	var params = {
			TableName:"Users",
			Key: {
				email : email
			},
			UpdateExpression: "set fullname = :fullname",
			ConditionExpression: "begins_with(fullname,:prefix)",
			ExpressionAttributeValues:{
		        ":fullname":fullName,
		        ":prefix":prefix
		    },
		    ReturnValues:"UPDATED_NEW"
		};
	
	docClient.update(params,callback);
}

Another feature is atomic counters. We can issue updates to a DynamoDB item and increase the attribute values. We will add an extra field called count. Also we will add another update function, which once called will update the field specified, but will also increase the counter attribute. Thus the counter attribute will represent how many times an update was performed on a specific item.

var addUpdateCounter = function(email,callback) {
	
	var docClient = new AWS.DynamoDB.DocumentClient();
	
	var params = {
			TableName:"Users",
			Key: {
				email : email
			},
			UpdateExpression: "set #counter = :counter",
			ExpressionAttributeNames:{
		        "#counter":"counter"
		    },
			ExpressionAttributeValues:{
		        ":counter":0
		    },
			ReturnValues:"UPDATED_NEW"
		};
	
	docClient.update(params,callback);
}

var updateAndIncreaseCounter = function(email,fullName,callback) {

	var docClient = new AWS.DynamoDB.DocumentClient();
	
	var params = {
			TableName:"Users",
			Key: {
				email : email
			},
			UpdateExpression: "set fullname = :fullname ADD #counter :incva",
			ExpressionAttributeNames:{
		        "#counter":"counter"
		    },
			ExpressionAttributeValues:{
		        ":fullname":fullName,
		        ":incva":1
		    },
		    ReturnValues:"UPDATED_NEW"
		};
	
	docClient.update(params,callback);
}

You can find the sourcecode on github.

Update DynamoDB Items with Java

On a previous post we proceeded into inserting items to DynamoDB using Java. DynamoDB also supports updating items.

We will use the Login table for the update examples.
When issuing an update you must specify the primary key of the item you want to update.

    public void updateName(String email,String fullName) {

        Map<String,AttributeValue> attributeValues = new HashMap<>();
        attributeValues.put("email",new AttributeValue().withS(email));
        attributeValues.put("fullname",new AttributeValue().withS(fullName));

        UpdateItemRequest updateItemRequest = new UpdateItemRequest()
                .withTableName(TABLE_NAME)
                .addKeyEntry("email",new AttributeValue().withS(email))
                .addAttributeUpdatesEntry("fullname",
                        new AttributeValueUpdate().withValue(new AttributeValue().withS(fullName)));

        UpdateItemResult updateItemResult = amazonDynamoDB.updateItem(updateItemRequest);
    }

We can proceed on more advanced statements using conditional updates. Conditional updates can help us in many cases such as handling concurrent updates.

We can achieve so by using plain expressions.

    public void updateConditionallyWithExpression(String email,String fullName,String prefix) {

        Map<String, AttributeValue> key = new HashMap<>();
        key.put("email", new AttributeValue().withS(email));

        Map<String, AttributeValue> attributeValues = new HashMap<>();
        attributeValues.put(":prefix", new AttributeValue().withS(prefix));
        attributeValues.put(":fullname", new AttributeValue().withS(fullName));

        UpdateItemRequest updateItemRequest = new UpdateItemRequest()
                .withTableName(TABLE_NAME)
                .withKey(key)
                .withUpdateExpression("set fullname = :fullname")
                .withConditionExpression("begins_with(fullname,:prefix)")
                .withExpressionAttributeValues(attributeValues);
        UpdateItemResult updateItemResult = amazonDynamoDB.updateItem(updateItemRequest);
    }

Or through by specifying attributes.

    public void updateConditionallyWithAttributeEntries(String email, String fullName, String prefix){

        Map<String,AttributeValue> key = new HashMap<>();
        key.put("email",new AttributeValue().withS(email));

        UpdateItemRequest updateItemRequest = new UpdateItemRequest()
                .withTableName(TABLE_NAME)
                .withKey(key)
                .addAttributeUpdatesEntry("fullname",new AttributeValueUpdate().withValue(new AttributeValue().withS(fullName)).withAction(AttributeAction.PUT))
                .addExpectedEntry("fullname",new ExpectedAttributeValue().withValue(new AttributeValue().withS(prefix)).withComparisonOperator(ComparisonOperator.BEGINS_WITH));

        UpdateItemResult updateItemResult = amazonDynamoDB.updateItem(updateItemRequest);
    }

Another feature is atomic counters. We can issue updates to a DynamoDB item and increase the attribute values. We will add an extra field called count. Also we will add another update function, which once called will update the field specified, but will also increase the counter attribute. Thus the counter attribute will represent how many times an update was performed on a specific item.

    public void addUpdateCounter(String email) {

        Map<String,AttributeValue> key = new HashMap<>();
        key.put("email",new AttributeValue().withS(email));

        UpdateItemRequest updateItemRequest = new UpdateItemRequest()
                .withTableName(TABLE_NAME)
                .withKey(key)
                .addAttributeUpdatesEntry("counter",new AttributeValueUpdate().withValue(new AttributeValue().withN("0")).withAction(AttributeAction.PUT));

        UpdateItemResult updateItemResult = amazonDynamoDB.updateItem(updateItemRequest);
    }

    public void updateAndIncreaseCounter(String email,String fullname) {

        Map<String,AttributeValue> key = new HashMap<>();
        key.put("email",new AttributeValue().withS(email));

        UpdateItemRequest updateItemRequest = new UpdateItemRequest()
                .withTableName(TABLE_NAME)
                .withKey(key)
                .addAttributeUpdatesEntry("fullname",new AttributeValueUpdate().withValue(new AttributeValue().withS(fullname)).withAction(AttributeAction.PUT))
                .addAttributeUpdatesEntry("counter",new AttributeValueUpdate().withValue(new AttributeValue().withN("1")).withAction(AttributeAction.ADD));

        UpdateItemResult updateItemResult = amazonDynamoDB.updateItem(updateItemRequest);
    }

You can find the sourcecode on github.

Scan DynamoDB Items with Node.js

On previous posts we covered how to query a DynamoDB database
Query DynamoDB Part 1
Query DynamoDB Part 2.

Apart from issuing queries DynamoDB also offers Scan functionality.
What scan does is fetching all the Items you might have on your DynamoDB Table.
Therefore scan does not require any rules based on our partition key or your global/local secondary indexes.
What scan offers is filtering based on the items already fetched and return specific attributes from the items fetched.

The snippet below issues a scan on the Logins table by adding filtering and selecting only the email field.

var scanLogins = function(date,callback) {

	var docClient = new AWS.DynamoDB.DocumentClient();

	var params = {
		TableName:"Logins",
		ProjectionExpression: "email",
	    FilterExpression: "#timestamp < :from",
	    ExpressionAttributeNames: {
	        "#timestamp": "timestamp",
	    },
	    ExpressionAttributeValues: {
	         ":from": date.getTime()
	    }
	};

	var items = []
	
	var scanExecute = function(callback) {
	
		docClient.scan(params,function(err,result) {

			if(err) {
				callback(err);
			} else {
				
				items = items.concat(result.Items);

				if(result.LastEvaluatedKey) {

					params.ExclusiveStartKey = result.LastEvaluatedKey;
					scanExecute(callback);				
				} else {
					callback(err,items);
				}	
			}
		});
	}
	
	scanExecute(callback);
};

Before using scan to an application we have to take into consideration that scan fetches all table items. Therefore It has a high cost both on charges and performance. Also it might consume your provision capacity.
It is better to stick to queries and avoid scans.

You can find the sourcecode on github.

Scan DynamoDB Items with Java

On previous posts we covered how to query a DynamoDB database
Query DynamoDB Part 1
Query DynamoDB Part2.

Apart from issuing queries DynamoDB also offers Scan functionality.
What scan does is fetching all the Items you might have on your DynamoDB Table.
Therefore scan does not require any rules based on our partition key or your global/local secondary indexes.
What scan offers is filtering based on the items already fetched and return specific attributes from the items fetched.

The snippet below issues a scan on the Logins table by adding filtering and selecting only the email field.

public List<String> scanLogins(Date date) {

        List<String> emails = new ArrayList<>();


        Map<String, String> attributeNames = new HashMap<String, String >();
        attributeNames.put("#timestamp", "timestamp");


        Map<String, AttributeValue> attributeValues = new HashMap<String, AttributeValue>();
        attributeValues.put(":from", new AttributeValue().withN(Long.toString(date.getTime())));


        ScanRequest scanRequest = new ScanRequest()
                .withTableName(TABLE_NAME)
                .withFilterExpression("#timestamp < :from")
                .withExpressionAttributeNames(attributeNames)
                .withExpressionAttributeValues(attributeValues)
                .withProjectionExpression("email");

        Map<String,AttributeValue> lastKey = null;

        do {

            ScanResult scanResult = amazonDynamoDB.scan(scanRequest);

            List<Map<String,AttributeValue>> results = scanResult.getItems();
            results.forEach(r->emails.add(r.get("email").getS()));
            lastKey = scanResult.getLastEvaluatedKey();
            scanRequest.setExclusiveStartKey(lastKey);
        } while (lastKey!=null);

        return emails;
    }

Before using scan to an application we have to take into consideration that scan fetches all table items. Therefore It has a high cost both on charges and performance. Also it might consume your provision capacity.
It is better to stick to queries and avoid scans.

You can find the sourcecode on github.