Part 3: All about Sitecore Serialization using Sitecore CLI

Series of Sitecore CLI blogs

You can check if you have the serialization plugin:

> dotnet sitecore plugin list

If plugin is not there, then install it by running:

> dotnet sitecore plugin add -n Sitecore.DevEx.Extensibility.Serialization

To get the information about serialization commands run:

> dotnet sitecore ser -h

You will get all the information about parameters like below:

Once setup of Sitecore CLI and Sitecore content serialization configuration is done, you are good to execute serialization of items.

First, login via interactive or non interactive flow(check link for more details):

Here, will go with non-interactive client login:

> dotnet sitecore login --authority https://<sitecore-identity-server> --cm http://<sitecore-instance> --allow-write true --client-credentials true --client-id <client-id> --client-secret <client-secret>

Login information will be saved in .sitecore/user.json.

Pull:

Serialize items from Sitecore to disk

You can figure out usage of individual commands with -h or –help command.

  • Pull all items as configured in *.module.json.
> dotnet sitecore ser pull

After pulling items, you will see .yml file corresponding to each item in sitecore in disk

  • Pull items from specific module only(with tags separated by comma ,For inserting tags, check – link)
> dotnet sitecore ser pull -i tags:[tagA,tagB]
  • Pull items from specific module only(with Module namespace)
> dotnet sitecore ser pull -i ModuleA ModuleB
  • Pull all items except certain modules with tags:
> dotnet sitecore ser pull -e tags:[tagA,tagB]
  • Pull all items except some modules(with Module namespace)
> dotnet sitecore ser pull -e ModuleA ModuleB
  • Pull items without integrity validation
> dotnet sitecore ser pull -s
  • Check the differences/changes in disk and Sitecore without actually pulling it
> dotnet sitecore ser pull --what-if

Validate

Ensures file system integrity of serialized items in disk and their paths

It performs the following checks on serialized items:

  • Invalid physical path.
  • Orphaned parent ID.
  • Non-included item.
  • Empty folder.
  • Duplicate item ID.
  • Non-unique path.

  • Run the validation on serialized items
> dotnet sitecore ser validate
  • Fix the issues found in validation above
> dotnet sitecore ser validate --fix

If the --fix command finds duplicate content items in your file system, it keeps the last updated one and deletes the oldest one.

Package

  • Create package of serialized items in disk
> dotnet sitecore ser pkg create -o FILE_PATH

Package with extension .itempackage will be created at the file path given in command.

Create package at root path location

> dotnet sitecore ser pkg create -o <Package name>
  • Install the created package in Sitecore
> dotnet sitecore ser pkg install -f FILE_PATH

Push

Serialize items from disk to Sitecore

Push commands can be executed just like pull commands.

  • Push all items as configured in *.module.json.
> dotnet sitecore ser push
  • Push items from specific module only(with tags separated by comma , for inserting tags, check – link)
> dotnet sitecore ser push -i tags:[tagA,tagB]
  • Push items from specific module only(with Module namespace)
> dotnet sitecore ser push -i ModuleA ModuleB
  • Push all items except certain modules with tags:
> dotnet sitecore ser push -e tags:[tagA,tagB]
  • Push all items except some modules(with Module namespace)
> dotnet sitecore ser push -e ModuleA ModuleB
  • Push items without integrity validation
> dotnet sitecore ser push -s
  • Check the differences/changes in disk and Sitecore without actually pushing it
> dotnet sitecore ser push --what-if

Explain

To check if specific path is present in any modules.
> dotnet sitecore ser explain --path "PATH"

It will return the module information in which mentioned “path” is present.

Watch

Pull the changed items automatically into file system.
  • Activate watch
> dotnet sitecore ser watch
Advertisement

Part 2: Sitecore Content Serialization Configuration(Sitecore CLI)

Series of Sitecore CLI blogs

After this, you will get following folders and files:

.sitecore folder will have schemas and user.json file.

schemas folder contains three schemas:

  • ModuleFile schema for each module configuration (<module name>.module.json)
  • RootConfiguration required in setting up sitecore.json
  • UserConfiguration schema used in user.json when we login into sitecore instance with interactive or non-interactive flow.

User.json saves the login information once the user logged in successfully via interactive or non-interactive flow. Do not commit this file in source control.

Configure sitecore.json

This file will be common to the project. We can configure following information in sitecore.json.

  • RootConfigurationFile.schema.json schema is added at the top
  • Modules: Add path of module or you can add a wildcard as given above to include all modules by itself based on architecture of project.
  • Plugins: plugins installed in this for serialization, index, publish, resource packaging and database are added here.
  • Serialization: It contains information about allowed max relative path length, module relative serialization path(where item ymls will be created), should it continue if serialization fails at some item, what fields need be excluded, etc.

There are several fields on items which do not contain important information and not required to be serialized. So you can add an array of the excluded fields with two properties: fieldId and description to Exclude property.

For Example:

"excludedFields": [
      {
        "fieldId": "badd9cf9-53e0-4d0c-bcc0-2d784c282f6a",
        "description": "__Updated by"
      }
]

NOTE:

-Excluding Revision fields from serialization can result in issues when publishing with the serialization command.
-You can also mention excluded fields in specific *module.json files to exclude from particular module items only.

Settings: It contains boolean entries to enable/disable telemetry and to enable the feature to check the Sitecore management services version compatibility.

Configure <module name>.module.json

  • Create module file with <module name>.module.json.
  • Add the Module schema at the top.

Update the path as per location of file.

  • After schema, add following:
{
  "namespace": "",
  "references": [""],
  "items": ""
}
From dev.sitecore.com
  • In items property, mention sitecore item’s include paths to sync items, descendants, children, etc. as given below:
items:"
"includes": [
        {
          "name": "<name>",
          "path": "<sitecore item path>",
          "scope": "<scope>",
          "allowedPushOperations": "<allowedPushOperations>"
          "rules": [
                    {
                        "path": "<sitecore item path>",
                        "scope": "<scope>"
                    }
        }
]"

Details about include properties:

From dev.sitecore.com
  • Order of the include paths is important. For example: templates, layouts, renderings, should be added first in include path list to avoid conflicts while syncing content items due to dependency on templates, layouts, etc.
  • You can also create a separate Base.module.json file for base items like templates, branches etc. and the add it as references(refer below code snippet) in rest of files. So in this way, we can order include paths inside the *.module.json and also set the hierarchy in which all module files need to be serialized.

  • You can set the relative path at the module level(refer below code snippet) inside items. This will override the defaultModuleRelativeSerializationPath given in sitecore.json.

  • You can also set the different exclude fields in different modules rather than setting same exclude fields for all modules.
...
items": {
        "includes": [
            {
                "name": "Apikey",
                "path": "/sitecore/system/Settings/Services/API Keys"
            },
            {
                "name": "Media",
                "path": "/sitecore/media library/my-first-jss-app"
            }
        ],
        "excludedFields":[
            {
	          "fieldID": "{EB504D1B-B612-4FFF-B239CA3BD7273D1B}",
		  "description": "FieldsForExclude1"
	    },
            {
		  "fieldID": "{3C2C061E-F61F-4DF6-89EA-0B7A56348737}",
		  "description": "FieldsForExclude2"
	     }
        ]
    }
  • Like the include paths of content items, you can exclude certain set of items with rules along with each include path. For example:
"items": {
  "includes": [
    {
      "name": "content",
      "path": "/sitecore/content/home",
      "rules": [
        {
           "path": "/products/legacy",
           "scope": "ignored"
        },
        {
           "path": "/products",
           "scope": "ItemAndDescendants",
           "allowedPushOperations": "createUpdateAndDelete"
        },
        {
           "path": "*",
           "scope": "ignored"
        }
      ]
    }
  ]
}

Details about rule properties:

From dev.sitecore.com

NOTE:

- Scope: Ignored, is only valid in configuring rules
- Sitecore CLI does not support duplicate item names and will break push/pull operations
  • In the initial serialization, you will require to sync everything. However, in proceeding serialization, you may require to serialize a module or couple of modules. You can mention tags in the individual module files and then use those tags with push and pull commands to sync more targeted changes rather than to sync all

Pull command with tags:

dotnet sitecore ser pull --include tags:[global-208]
  • You can also serialize role by adding roles property in module.json. The roles property is an array that consists of role predicate items with two properties: domain (Sitecore role domain) and pattern (a regex pattern to determine specific roles to include under the domain). For example:
{
    ...
    "items": {
        ...
    },
    "roles": [
      {
        "domain": "sitecore",
        "pattern": "Developer"
      },
      {
        "domain": "custom",
        "pattern": "Role*"
      },
      {
        "domain": "extranet",
        "pattern": "^MySite.*$"
      }
    ]
}

Part 1: Getting started with Sitecore CLI

Series of Sitecore CLI blogs

Following are the steps to install and use Sitecore CLI for your project:

Pre requisites:

Install  .NET Core 

Sitecore Instance

Working Sitecore identity

Steps to install:

  • Install Sitecore Management services package(few dlls and configs) on your sitecore instance. Make sure to check compatibility from here.
  • Open Powershell in admin mode
  • Run following commands in your root directory:

> dotnet new tool-manifest
> dotnet nuget add source -n Sitecore https://sitecore.myget.org/F/sc-packages/api/v3/index.json
> dotnet tool install Sitecore.CLI 
Use the -g option when running the install command to install CLI globally. However, it is not recommended because different instances may need different version of Sitecore CLI
  • Initialize a new project
> dotnet sitecore init
  • Install the required Publishing and Serialization plugins
> dotnet sitecore plugin add -n Sitecore.DevEx.Extensibility.Serialization
> dotnet sitecore plugin add -n Sitecore.DevEx.Extensibility.Publishing

You can check the list of plugins installed by using following command:

> dotnet sitecore plugin list

After this, your folder will look like following:

To verify the sitecore CLI is successfully installed and working(-h for help)

> dotnet sitecore -h

Login to Sitecore with Sitecore CLI:

Sitecore CLI supports two flows of authentication and authorization.

  • An interactive user login, using a device code flow. Please follow below steps for interactive login:
> dotnet sitecore login --authority https://<sitecore-identity-server> --cm https://<sitecore-instance> --allow-write true

This will navigate you to browser to authorize and modify ~/.sitecore/user.json with access tokens. Make sure not to commit user.json as it contains sensitive information. After this, you are good to serialize items.

  • A non-interactive client login, using a client credentials flow. This is used by clients such as Continuous Integration servers. Please follow below steps for non- interactive login:

Make configuration changes:

create a file named Sitecore.IdentityServer.DevEx.xml and add following:

<?xml version="1.0" encoding="utf-8"?>
<Settings>
  <Sitecore>
    <IdentityServer>
      <Clients>
        <!-- used to authenticate servers with client id and client secret -->
        <CliServerClient>
            <ClientId>SitecoreCLIServer</ClientId>
            <ClientName>SitecoreCLIServer</ClientName>
            <AccessTokenType>0</AccessTokenType>
            <AccessTokenLifetimeInSeconds>3600</AccessTokenLifetimeInSeconds>
            <IdentityTokenLifetimeInSeconds>3600</IdentityTokenLifetimeInSeconds>
            <RequireClientSecret>true</RequireClientSecret>
            <AllowOfflineAccess>false</AllowOfflineAccess>
            <AllowedGrantTypes>
                <!--
                    client_credentials authenticates with client ID and client secret
                    which is good for CI, tools, etc. However, it's not tied to a USER,
                    it's tied to a client ID.
                -->
                <AllowedGrantType1>client_credentials</AllowedGrantType1>
            </AllowedGrantTypes>
            <ClientSecrets>
                <!--<ClientSecret1>SUPERLONGSECRETHERE</ClientSecret1>-->
            </ClientSecrets>
            <AllowedScopes>
                <!-- this is required even if not a 'user' for Sitecore to like us -->
                <AllowedScope1>sitecore.profile.api</AllowedScope1>
            </AllowedScopes>
        </CliServerClient>
      </Clients>
    </IdentityServer>
  </Sitecore>
</Settings>

Add client id and client secret(max length = 100) in this file. Keep this file in Config folder of Sitecore Identity Server.

create another file Sitecore.Owin.Authentication.ClientCredentialsMapping.
config
 containing the following:

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
  <sitecore role:require="Standalone or ContentDelivery or ContentManagement">
    <federatedAuthentication>
      <identityProviders>
        <identityProvider id="SitecoreIdentityServer" type="Sitecore.Owin.Authentication.IdentityServer.IdentityServerProvider, Sitecore.Owin.Authentication.IdentityServer" resolve="true">
          <transformations hint="list:AddTransformation">
            <transformation name="admin-ify client credentials users" type="Sitecore.Owin.Authentication.Services.DefaultTransformation, Sitecore.Owin.Authentication">
              <sources hint="raw:AddSource">
                <claim name="client_id" value="SitecoreCLIServer" />
              </sources>
              <targets hint="raw:AddTarget">
                <claim name="name" value="sitecore\superuser" />
                <claim name="http://www.sitecore.net/identity/claims/isAdmin" value="true" />
				<claim name="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress" value="" />

              </targets>
              <keepSource>true</keepSource>
            </transformation>
          </transformations>
          
        </identityProvider>
      </identityProviders>
    </federatedAuthentication>
  </sitecore>
</configuration>

Add email id(same client id as first file) in email address claim value. And keep it in App_Config/Include/ folder of Sitecore Content management server.

Do the IIS Reset.

Open powershell in admin mode.

Now login:

> dotnet sitecore login --authority https://<sitecore-identity-server> --cm http://<sitecore-instance> --allow-write true --client-credentials true --client-id <client-id> --client-secret <client-secret>

Login information will be saved in .sitecore/user.json

Next step would be Sitecore serialization configuration- structure, rules and items pulled depends decisions taken in defining the configurations.

How you want the items to be pulled? you want to opt for site wise folders, component wise folders, page wise folders, etc. What fields should be excluded while performing pull and push of items?

Everything depends on how you plan to do Sitecore serialization configuration. Please find the details in next blog.

Sitecore CLI login without Identity server

Find the listener by running dotnet Sitecore.IdentityServer.Host.dll in Identity server root folder:

Now login with listener got in above step: https://localhost:5000 or http://localhost:5000 as authority parameter (bypassing identity server) in login command:

dotnet sitecore login --authority https://<sitecore-identity-server> --cm https://<sitecore-instance> --allow-write true