1. Multi-Cloud Support: Runs on any cloud, public or private, and support on most on-premises infrastructure platforms.
2. Container ready: Built-in support for standard container formats like Docker, making it easy to build, compose, deploy and move your workloads easily.
3. Built-in security and scalability: Enterprises can define fine-grained policies for access management, scaling, and other life cycle management operations that encourages the platform natively support them. Its built-in workflow manager can be used to embed approval steps where required to provide an additional layer of controls.
4. Multi-Tenancy: Service providers and the Enterprise can achieve clear isolation and assure autonomy by enforcing policies at the tenant, project, user, or service levels.
5. Self-governance: Built on the core principle of providing pervasive governance to ensure consistent operational behavior from availability, performance, compliance, and a security point of view is provided.
6. Blueprint support: Developers and architects can group services into a logical entity called Blueprint with common scaling policies and access controls. Once modeled these blueprints can be used to deploy the group of services with a single click and manage the life cycle as per the policies specified.
7. DevOps Console: Microservices-based applications are dynamic in nature and Macaw DevOps console can help developers and operations team deploy and monitor their runtime behavior for easy troubleshooting.
8. Application Modernization: Traditional applications can be transformed without any major disruption to embrace the principle of cloud-native applications. Macaw allows gradual migration to Microservices by leveraging the middleware support or shadow services to maintain integration with the legacy systems.
9. Curated stack of open source technologies: The platform is built using the best of breed open source technologies including Kafka, Zookeeper, Spark, Cassandra, Docker, Swarm, Elastic Search among others. These technologies provide the scalability, multi-tenancy, and robustness enterprises demand. Further enhancement with automation and governance features simplify usage for DevOps functions.

Service Infrastructure: Macaw’s set of core capabilities that provide a solid foundation of both services and platform.

• Message Bus: Distributed Kafka & Zookeeper over SSL running on multiple container nodes, used dually for internal and external messaging between microservices.
• Logging & Search: Logging as a service provides centralized logging capabilities leveraging Elasticsearch.
• DBaaS: Utilize Cassandra and MySQL database as services. Macaw covers the DB lifecycle, including init data, schema loading, loading DB drivers, and automatically mapping and building dependencies for service/container images.
• Message Correlator: All user-generated API request chains are fully traceable and correlated through message ID and can be viewed in the DevOps console events widget. Serves effectively for troubleshooting plus acts as an analytics tool to build social service graphs.
• Analytics: Yields essential intelligence about service operational behavior like service performance, life cycle events, usage stats, etc. through native analytics as well as pluggable external analytics providers like Apache Spark and more.

Macaw Runtime: Macaw platform core runtime is polyglot with support for multiple languages including Java, Javascript, Python etc.

Core Services: Macaw platform capabilities (i.e. service discovery, registry, provisioner, etc.) that are integral in a Microservices-based architecture.

• Service Registry: All Macaw microservices register with a centralized Service Registry regarding service listing, service lookups, and interactions among services or with external clients.
• Service Provisioner: Provisions any service on a Macaw node based on native placement and a load balancing algorithm. Configurable also to provision service on external container clusters through resource schedulers like Kubernetes or Docker Swarm
• API Gateway: Aggregates all Macaw microservices APIs as RESTful API endpoints and provides API browser, invocation capabilities through the DevOps console.
• Blueprints: Collection of related Macaw microservices along with their operational, placement, and deployment policies together constitute Macaw blueprint.

Advanced Services: Premium services consisting of additional applications that run on Macaw platform.

• Locker: A secure way to store and access credentials using multi-level encryptions and advanced security principles similar to that of popular cloud providers.
• Identity: Identity services to other Macaw microservices equipped with knowledge of all users, orgs, tenants.
• Service Governance: Service migration-based on load or behavior, service access governance-based on policies, approval injections at platform level are some of the capabilities provided by Service Governance
• Notification Manager: Responsible for sending notifications to interested clients about service lifecycle events.

Multi-Tenancy: Supports multiple tenants to operate on a single instance of Macaw platform through support for tenants, projects, users and roles.

Service Operations: Macaw catalog services that integrate with external applications to enhance the operational behavior of macaw applicationss.

• APM: Allows Macaw microservices to be monitored through existing APM tools by serving Macaw service data to target APM tools.
• Ticketing: Allows Macaw microservices to create/update/delete tickets in a target ticketing system-based on certain service lifecycle events.
• Shadow Services: Integration with legacy systems and external applications to bring them into Macaw’s operational environment to streamline the application modernization process.

Images & Repositories: Macaw microservices are packaged as Docker containers. MDR (Meta Data Repository) holds the service blueprints, Metadata information with available Docker tags for a specific service. Docker registry holds service container images.

CI/CD Plugins: Integration with CI/CD tools for accelerated release and feature velocity.

IDE Plugins: Improves developer productivity by providing plug-ins and project configuration files for popular IDEs like Eclipse.

Developer Toolset: A set of essential tools to accelerate service development and increase developer productivity.

• Modeling: Capture API model in YANG or JSON and Swagger documentation.
• Codegen: Generates Java artifacts such as public interfaces and class stubs.
• Build: Build entire Microservices and automatically build service and container images.
• Publish: CLI utilities to easily publish a Microservice to repository using an image tag.

Access and Administrative Interfaces: Macaw can be accessed and administered in multiple ways.

• DevOps Console: A slick web-based HTML 5 interface for DevOps professionals to operate Macaw microservices and the  platform.
• Macaw CLI: CLI-based interface for platform and service operations.
• API Gateway: API-based service interactions and administrations suitable for automation and integrations with external tools.

Infrastructure: Infrastructure integration, management and orchestration layers allow Macaw to operate on various IaaS/Cloud platforms including BareMetal, VM, Cloud, or a developer laptop.

Before proceeding with installation, refer to the details listed below to directly understand Macaw’s terminologies and supported installation configurations.

Terminologies

In the installation documentation, the following terminologies are used. Comprehension of the terminologies, what they mean, and what role they play in the overall Macaw platform is crucial to the installation process.

Platform VM and Service VM can be deployed as OVFs. In preparation to deploy the OVFs, keep the IP Addresses, Gateway, Netmask, DNS, NTP, and Hostname details for each VM handy. It is extremely important to have DNS mapping for the hostnames and all nodes to be timed synced with NTP.

Note: Macaw delivered OVFs are compatible with VMware vSphere 5.1 and above.

Prerequisites to deploy OVF’s are as follows.

Note: For the rest of the document, the nodes will be referred to as displayed here (using the following convention to qualify the context):

Macaw Platform VM – platform.domain.com
Macaw Service VM – service.domain.com

3.1 Two VMware OVFs are delivered as a part of deliverables:

1. Platform-VM-0.9.4.zip (Macaw Platform VM in OVF Format)
2. Services-VM-0.9.4.zip (Macaw Services VM in OVF Format)

3.2 Download of OVFs

Download the OVFs using the following URLs (contact POC or Support contact at support@www.macaw.io) onto a machine where the vSphere client is available (e.g Windows box where the VMWare vSphere client/environment will be accessible to deploy the above downloaded OVFs.).

Platform OVF: <URL>
Services OVF: <URL>

3.3 Extract or Unzip the above downloaded OVF’s using standard unzip utility ( 7z utility ) as depicted in following screen-shots.

3.4  Deploy OVFs using VMware vSphere Client (using wizard):

The above step will prompt wizard to deploy the OVF’s. Please make sure you have all the prerequisites before deployment starts

Accept license window and follow the steps,

We have three types of deployments in OVF Configurations Small, Medium and Large .We need to select based on requirements,configuration for production is Large.

Select the “deploy OVF template” option and wizard will walk through the deployment process. Please make to have the following details ready before deploying the VMs as described in the prerequisite section of the document.

• Cluster: where to deploy these VMs
• Storage: to use for deploying VMs
• Network: to use for deploying VMs (Port Group details reachable to network)
• Hostname, IP Address, Gateway etc) as wizard will prompt to enter those values during the deployment process

3.5  Power On VMs

Once the above step is completed (deployment of OVF templates into vSphere environment is completed using wizard), power on the VMs as shown below.

Once the VM’s are powered on, make sure to access the above VM(s) using the DNS/IP entries (via ping or ssh to connect to the above deployed and configured VMs over the network) that were provided by the user to the VMs during deployment.

3.6  Login Verification of deployed VMs (OVFs):
Verify the login via SSH client of choice to Platform VM and Service VM Nodes with the below credentials.

E.g :-
————————————————————————————-
ssh macaw@platform.domain.com <Enter>
User: macaw<Enter>
Password: appd1234<Enter>
————————————————————————————-

The above will login into Macaw Platform / Service VM shell.

Note: Another user ‘macaw’ belonging to group ‘macaw’ is also defined under whom the infrastructure services would be running. This user doesn’t have any password.

Once the above steps are done, move to the platform installation section.

• Login to your AWS EC2 cloud and navigate to the EC2 dash board.

• Go to the AMIs and look for macaw Platform and Service Instances by typing ‘Macaw’ in the search filter like shown below. These are public AMIs currently. Contact POC or support contact at support@www.macaw.io to get further details on the AMI.
• Refer to the AWS link on how to find public AMIs. Refer to the below screen on how search.
• Select the Public Images view under the AMIs.

• Select the macaw Platform Instance and click on Launch.

• Chose an Instance Type.

Note: Production installation of macaw Platform AMI requires minimum 8 vCPUs and 24 GIB of Memory.Select m4.2xlarge as the instance type for 
production installations and click on Configure Instance Details. For POCs you can chose a smaller type like t2.large or above. Minimum 
8GB RAM is required for macaw platform.

• The below settings in the next page are specific to your AWS environment. Brief explanation is provided below with some details.

Number of Instances: Select 1

Purchasing Option: <This is specific to your AWS Environment. If not sure, please leave it unchecked>

Network: This is specific to your AWS environment. This is the VPC for AWS region your are operating in. 
Check with your admin on what VPC to use.

Subnet: This is also specific to your AWS Environment. Try selecting a subnet for which public IP is 
assigned, so that you can access the machines from outside.

Auto-assign Public IP: If you are not sure of Public IP assignment for the above subnet, 
you can select this to Enable.

Placement group: <Check with AWS Admin. If not sure, leave default>

IAM Role: <Check with AWS Admin. If not sure, leave default>

Shutdown Behavior: <Check with AWS Admin. If not sure, leave default>

<Rest All Options Leave Default>

• Adding Storage

macaw Platform AMI runs disk intensive programs like kafka/zookeeper/cassandra. Hence it is absolutely necessary to have separate disks hosting the relevant data for these programs. The AMI also supports minimal installation mode, where extra disks are not needed and the user can select default disks with the AMI and just change the size.

Non Production Disk Configuration

Production Disk Configuration

Note: The Platform AMI auto mounts all these additional volumes.

Once the storage is configured, please click “Next: Tag Instance”

• Tag Instance

Provide a name to the platform instance and then click “Configure Security Group”

• Configure Security Group

Select the option “Select an existing security group” and then select the Platform Instance Security Group you defined earlier and click on “Review and Launch”.

• Review your selections and then click launch.
• Next your will be prompted for selection of keys. This is where AWS will let your chose an exiting key or to create a key pair and program your instance with the public key of the key pair. You are supposed to be holding the private key of the pair. If you lose the private key, it cannot be downloaded again. AWS only lets you download the key pair once when it is created.

Note: You can chose to create a new pair and use this for all macaw related AWS instances. Make sure to download the private key and store it securely.

• Click Launch and wait for instance to be available.

Launching Service Instance

If you need service instance as well, follow similar instructions on the launch service instance as well. The service instance AMI doesnt require any additional disks. For the pre-populated disks, chose the sizes as described above for the platform instance (These base mandatory disks follow the same recommendations). When selecting the security group, make sure to select the one which is defined for the Services.

Viewing Instance Details

• Click on the Instances in the EC2 dashboard view.
• In the list shown, select either Macaw platform or Service Instance.

The Public DNS/IP can be used to access the instance using SSH.

Accessing Instance

macaw AMI instances are based on standard x86_64 CentOS7. The AMIs are programmed with centos user and the public key of the key pair you have used during the launch of the AMIs, is programmed for this user. You can use the private key of the pair to login.

ssh -i <path to the private key> macaw@<Platform Instance Public IP>

ssh -i <path to the private key> macaw@<Service Instance Public IP>

Refer to the below AWS links for accessing the instance using the key.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html

The macaw AMIs are programmed with macaw user. Password based SSH is enabled for this specific user. The default password is “macaw”. It is strongly recommended to changed the password on all the instances.

ssh macaw@<Platform Instance Public IP>

ssh macaw@<Service Instance Public IP>

You can use putty or any other standard windows based SSH client to access the Instance using SSH. From linux, SSH client is standard utility and should be available on all flavors of Linux.

sudo pip install <Location or HTTP link of macawcli tar Package>

Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology. Vagrant is controlled by a single consistent workflow to help maximize the productivity and flexibility of the user and their team. For more details on Vagrant Installation, refer to the Vagrant documentation.

Macaw platform can be installed locally on a Windows desktop/MAC running Vagrant/Virtual Box. The below documentation assumes that working installation of Vagrant and Virtual box on the system are in place. The recommended version of Vagrant and virtual box are listed below.

Vagrant Download Link     |    Virtual Box Download Link

Note: To install Vagrant on Windows , openssh for Windows  must also be installed as a prerequisite.

$vagrant version
Installed Version: 1.8.7
Latest Version: 1.8.7
You're running an up-to-date version of Vagrant!
$ VBoxManage --version
5.1.10r112026

Download the macaw Box Image

1. Get the http link for the box image. Please contact the Macaw support team to help provide an http link for the downloadable box.
2. Execute the below command to download the macaw box image terminal window (Mac) or a Command Window (Windows).

$ vagrant box add <HTTP Link for macaw Box>
box: Loading metadata for box '<HTTP Link>'
box: Adding box 'macaw' (v1.0.0) for provider: virtualbox
box: Downloading: <HTTP Link>/boxes/macaw-1.0.0.box
box: Box download is resuming from prior download progress
box: Calculating and comparing box checksum...
box: Successfully added box 'macaw' (v1.0.0) for 'virtualbox'!

Note: Depending on network speed, the download may take some time.

3. Verify if the box has been downloaded properly  using the below command.

$ vagrant box list
centos/7 (virtualbox, 1609.01)
macaw    (virtualbox, 1.0.0)

Running the macaw Box Image

1. Depending on the version of the operating system running, create a folder with any name like ‘macawplatform’. Change to that directory.
2. Run the command below from either terminal window (Mac) or a Command Window (Windows)

$ mkdir macawplatform
$ cd macawplatform/
$ vagrant init macaw
A `Vagrant file` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrant file as well as documentation on
`vagrantup.com` for more information on using Vagrant.

You need to define an environment variable MACAW_PLATFORM_VERSION=<platform Version> For Example, if you are using 0.9.4, please provide like below.

$ export MACAW_PLATFORM_VERSION=0.9.4

Note: For Windows you can use the below. Make sure to exit the Powershell Window and re-open after executing this. 
The environment variable doesnt take into effect in the current powershell window.

[Environment]::SetEnvironmentVariable("MACAW_PLATFORM_VERSION", "0.9.4", "User") 

$ vagrant up

Bringing machine 'default' up with 'virtualbox' provider...

==> default: Importing base box 'macaw'...

==> default: Matching MAC address for NAT networking...

==> default: Checking if box 'macaw' is up to date...

==> default: Setting the name of the VM: macaw-platform-vm-0.9.4

==> default: Fixed port collision for 22 => 2222. Now on port 2200.

==> default: Clearing any previously set network interfaces...

==> default: Preparing network interfaces based on configuration...

    default: Adapter 1: nat

    default: Adapter 2: hostonly

==> default: Forwarding ports...

    default: 22 (guest) => 2200 (host) (adapter 1)

==> default: Running 'pre-boot' VM customizations...

==> default: Booting VM...

==> default: Waiting for machine to boot. This may take a few minutes...

    default: SSH address: 127.0.0.1:2200

    default: SSH username: macaw

    default: SSH auth method: private key

    default: Warning: Remote connection disconnect. Retrying...

    default: Warning: Remote connection disconnect. Retrying...

    default: Warning: Remote connection disconnect. Retrying...

    default: Warning: Authentication failure. Retrying...

    default: 

    default: Vagrant insecure key detected. Vagrant will automatically replace

    default: this with a newly generated keypair for better security.

    default: 

    default: Inserting generated public key within guest...

    default: Removing insecure key from the guest if it's present...

    default: Key inserted! Disconnecting and reconnecting using new SSH key...

==> default: Machine booted and ready!

==> default: Checking for guest additions in VM...

==> default: Configuring and enabling network interfaces...

==> default: Mounting shared folders...

    default: /vagrant => /Users/ravjanga/Documents/0.9.4.b7

==> default: Running provisioner: macaw Registry Login (shell)...

    default: Running: script: docker login to registry.macaw.io

==> default: Login Succeeded

==> default: Running provisioner: macaw Infra (shell)...

    default: Running: script: Downloading macaw infra components

==> default: Pulling registry.macaw.io/zookeeper:macaw-v0.9.4

==> default: Pulling registry.macaw.io/kafka:macaw-v0.9.4

==> default: Pulling registry.macaw.io/cassandra:macaw-v0.9.4

==> default: Pulling registry.macaw.io/elasticsearch:macaw-v0.9.4

==> default: Pulling registry.macaw.io/mysql:macaw-v0.9.4

==> default: Pulling registry.macaw.io/haproxy:macaw-v0.9.4

==> default: Pulling registry.macaw.io/macaw-tomcat:macaw-v0.9.4

==> default: Pulling registry.macaw.io/redis:macaw-v0.9.4

==> default: Running provisioner: macaw Platform (shell)...

    default: Running: script: Downloading macaw platform components

==> default: Pulling registry.macaw.io/service-registry:macaw-v0.9.4

==> default: Pulling registry.macaw.io/notification-manager:macaw-v0.9.4

==> default: Pulling registry.macaw.io/identity:macaw-v0.9.4

==> default: Pulling registry.macaw.io/service-provisioner:macaw-v0.9.4

==> default: Pulling registry.macaw.io/user-preferences:macaw-v0.9.4

==> default: Pulling registry.macaw.io/console-ui-webapp:macaw-v0.9.4

==> default: Pulling registry.macaw.io/macaw-dbinit:macaw-v0.9.4

==> default: Running provisioner: Installing macawcli Tool (shell)...

    default: Running: script: Downloading macawcli

==> default: Installing macaw CLI packge from https://macaw-amer.s3.amazonaws.com/tools/macawcli-0.9.4.tar.gz

==> default: Macawcli Tool package Installed Successfully

==> default: Running provisioner: IP/Hostname Mapping (shell)...

    default: Running: script: IP/Hostname Mapping

==> default: IP/Hostname Mapping in /etc/hosts successful

==> default: Machine 'default' has a post `vagrant up` message. This is a message

==> default: from the creator of the Vagrantfile, and not from Vagrant itself:

==> default: 

==> default: Congratulations!! You are now ready to login to the Macaw Platform VM

Note: The vagrant Macaw box is customized with the properties listed here. Overriding the defaults can be done with a few supported environment variables.

1. Memory default is set to 8GB.
2. CPU count is set to 2.
3. Network is set to private_network with hardcoded IP address of 192.168.33.10. This IP is reachable from the Host only. The VM will be able to access the internet. The documentation of further steps assume this IP address in installing the Macaw platform.

Verifications

Once the above command is successful, the user can login to the Macaw platform VM and be able to check a few things to confirm if the installation was performed smoothly.

$ vagrant ssh
Last login: Sat Nov 12 23:26:12 2016
[vagrant@localhost ~]$ ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:40:c1:5d brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 85741sec preferred_lft 85741sec
    inet6 fe80::a00:27ff:fe40:c15d/64 scope link 
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:97:8d:6c brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.10/24 brd 192.168.33.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe97:8d6c/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:cb:86:21:94 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
[vagrant@localhost ~]$ ping 8.8.8.8 -c 5
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=63 time=18.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=63 time=16.1 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=63 time=19.6 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=63 time=297 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=63 time=14.5 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 14.558/73.298/297.207/111.969 ms
[vagrant@localhost ~]$
[vagrant@localhost ~]$ nproc

2
[vagrant@localhost ~]$ free -m

              total        used        free      shared  buff/cache   available
Mem:           5807         148        4040           8        1617        5426
Swap:          2047           0        2047

[vagrant@localhost ~]$ macaw -v
Version: macawcli-0.9.4

Note: The Macaw tool version can be different from what is shown above.

After successful execution of the above steps, move forward with the platform installation. Follow the link for Macaw platform installation. For installing the Macaw platform, during setup the user would need to provide FQDN of the platform host. In this case, IP address 192.168.33.10 can be used for the platform IP as well as the service host IP.

Macaw platform is supported on generic 64 Bit Linux OS which supports Docker. The steps below provide an automated way of installing the necessary packages needed for Macaw platform and also manual steps if required. This installed script is only supported on CentOS7 64-bit OS and version 3.10 or higher of the Linux kernel.

Prerequisites

• Check for CentOS Version. It should be minimum 7.

$ more /etc/centos-release
CentOS Linux release 7.3.1611 (Core)
$ uname -r
3.10.0-514.16.1.el7.x86_64

• Check availability of the command curl. This command is available on most linux installations by default.

$ which curl
/usr/bin/curl

• DNS and Hostname

Make sure the platform host is configured with the fully qualified host name with DNS resolution. It is highly recommended to have proper DNS entry for the host name and avoid using the/etc/hosts mapping of the host name to IP.

If DNS is not available in your network, you can use the IP Address of the host. This is not recommended in production deployments. In this have the below entry in your /etc/hosts file on the platform Host.

<Platform HostIP> <Platform Hostname>

• Firewall Settings

If firewall is disabled, the user can ignore this pre-requisite. If firewall is enabled on the Linux Host, then for platform installation the ports/services listed below need to be opened up.

firewall-cmd --list-ports
443/tcp 80/tcp 8181/tcp 4000/tcp 2181/tcp 9200/tcp 3306/tcp 8443/tcp 8637/tcp 9042/tcp 9300/tcp 9092/tcp 7000/tcp 6379/tcp 5000/tcp 9160/tcp

firewall-cmd --list-services
rpc-bind nfs ssh mountd

Install with Automated Script

• Log into the machine as a user who has password-less sudo permissions or root privileges.
• Make sure the existing packages are up-to-date.

sudo yum update

• If yum updates the kernel, it is recommended to restart the machine.
• Run the Macaw installation script.

Note: When the user registered for the Macaw software download, the confirmation email would have a link to the MacawInstall script, a shell script.

$ bash <(curl -s https://s3.amazonaws.com/macaw-amer/tools/macawInstall.sh) --role <Role of the VM/Host>

Note: Once the installation is done, make sure logout and login back again so that group permissions get updated for the user.

To Prepare a Platform VM/Host pass --role platform
To Prepare a Service VM/Host pass --role service

• Install the Macaw CLI Python package. The location of the macaw CLI package should be part of the Macaw download/setup email.

sudo pip install <URL / Location of the macawcli Package>

• Verify the Macaw tool by issuing the command “macaw –version”

Once the installation/verification is done, the user can move to the Platform Installation section.

Manual Installation Steps – Platform VM/Host

If installer is not used or cannot be used in your environment, the below steps can be executed manually.

• Yum Update

Update the OS to get latest security patches.

yum update -y

Note: Please reboot the system after doing this operation.

• NFS Client Utils
  

Macaw platform mandates NFS shared mount between platform and service hosts. To be able to mount NFS,  the user needs to install NFS client utilitiess.

sudo yum install -y nfs-utils
sudo systemctl enable rpcbind
sudo systemctl start rpcbind

• Install autofs

Autofs module is used to mount NFS volumes.

sudo yum install -y autofs
sudo sed -i 's/.*auto.misc.*/\/-      \/etc\/auto.macawnfs/g' /etc/auto.master
sudo touch /etc/auto.macawnfs
sudo systemctl enable autofs
sudo systemctl start autofs

• Install macaw Trusted certificate

sudo wget -q -P /etc/pki/ca-trust/source/anchors/ https://s3.amazonaws.com/macaw-amer/thirdparty/__macaw_io.ca-bundle
sudo update-ca-trust

• Install Docker

Follow the docker installation instructions at the below link.

https://docs.docker.com/engine/installation/linux/centos/

Below are simple instructions to install docker 1.13.1 release. It is highly recommended to follow production installation guidelines for Docker.

sudo yum-config-manager --add-repo https://s3.amazonaws.com/macaw-amer/thirdparty/docker.repo
sudo yum makecache fast
sudo yum -y install docker-engine-1.13.1 docker-engine-selinux-1.13.1
sudo systemctl daemon-reload
sudo systemctl enable docker
sudo systemctl start docker

Once the docker installation is done, please perform the below necessary post-installation steps.

sudo usermod -aG docker <current user ID>

Please logout of the current session and login again so that the groups permissions is taken into account.

Note: Current user ID is the user who would be executing the Macaw installation later as well. The user should be using this user ID across the Macaw installation steps.

• Install JAVA

Java 1.8 must be installed under /opt/java

sudo yum install -y wget
/bin/rm -rf /tmp/java-linux.tar.gz
sudo mkdir -p /opt/java
sudo wget -q -O /tmp/java-linux.tar.gz https://s3.amazonaws.com/macaw-amer/thirdparty/jdk-8u102-linux-x64.tar.gz
sudo tar xf /tmp/java-linux.tar.gz -C /opt/java --strip-components 1
/bin/rm -rf /tmp/java-linux.tar.gz

Now update the PATH of the current user so that Java is found in the PATH. Add the below line to the ~/.bash_profile

PATH=/opt/java/bin:$PATH

export PATH

Note: Make sure to logout and login back. Verify that Java is found in the PATH by executing

$ which java
/opt/java/bin/java

$ java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

• Enable NFS Server

If the user doesn’t have an external NFS server, they can install and enable NFS on the platform host. Once installed, they can NFS export a mount from the platform host and all the service hosts can NFS mount the directory. The exporting/mounting is all done as part of the Macaw setup.

sudo yum install -y nfs-utils libnfsidmap
sudo systemctl enable rpcbind
sudo systemctl enable nfs-server
sudo systemctl start rpcbind
sudo systemctl start nfs-server
sudo systemctl start rpc-statd
sudo systemctl start nfs-idmapd

sudo yum install -y mysql wget

python --version

curl --silent --show-error --retry 5 https://bootstrap.pypa.io/get-pip.py | sudo python
sudo pip --version

sudo yum install -y gcc kernel-devel libffi-devel python-devel openssl-devel

sudo pip install <Full Local or Remote path location of macawcli.tar.gz>

macaw -v

sudo yum remove -y gcc kernel-devel libffi-devel python-devel openssl-devel

Once the installation is done, the user can move to the Platform Installation section.

Manual Installation Steps – Service VM/Host

If installer is not used or cannot be used in your environment, the below steps can be executed manually.

• Yum Update

Update the OS to get latest security patches.

yum update -y

Note: Please reboot the system after doing this operation.

• NFS Client Utils
  

Macaw platform mandates NFS shared mount between platform and service hosts. To be able to mount NFS,  the user needs to install NFS client utilities.

sudo yum install -y nfs-utils
sudo systemctl enable rpcbind
sudo systemctl start rpcbind

• Install autofs

Autofs module is used to mount NFS volumes.

sudo yum install -y autofs
sudo sed -i 's/.*auto.misc.*/\/-      \/etc\/auto.macawnfs/g' /etc/auto.master
sudo touch /etc/auto.macawnfs
sudo systemctl enable autofs
sudo systemctl start autofs

• Install macaw Trusted certificate

sudo wget -q -P /etc/pki/ca-trust/source/anchors/ https://s3.amazonaws.com/macaw-amer/thirdparty/__macaw_io.ca-bundle
sudo update-ca-trust

• Install Docker

Follow the docker installation instructions at the below link.

https://docs.docker.com/engine/installation/linux/centos/

Below are simple instructions to install docker 1.13.1 release. It is highly recommended to follow production installation guidelines for Docker.

sudo yum-config-manager --add-repo https://s3.amazonaws.com/macaw-amer/thirdparty/docker.repo
sudo yum makecache fast
sudo yum -y install docker-engine-1.13.1 docker-engine-selinux-1.13.1
sudo systemctl daemon-reload
sudo systemctl enable docker
sudo systemctl start docker

Once the docker installation is done, please perform the below necessary post-installation steps.

sudo usermod -aG docker <current user ID>

Please logout of the current session and login again so that the groups permissions is taken into account.

Note: Current user ID is the user who would be executing the Macaw installation later as well. The user should be using this user ID across the Macaw installation steps.

• Install JAVA

Java 1.8 must be installed under /opt/java

sudo yum install -y wget
/bin/rm -rf /tmp/java-linux.tar.gz
sudo mkdir -p /opt/java
sudo wget -q -O /tmp/java-linux.tar.gz https://s3.amazonaws.com/macaw-amer/thirdparty/jdk-8u102-linux-x64.tar.gz
sudo tar xf /tmp/java-linux.tar.gz -C /opt/java --strip-components 1
/bin/rm -rf /tmp/java-linux.tar.gz

Now update the PATH of the current user so that Java is found in the PATH. Add the below line to the ~/.bash_profile

PATH=/opt/java/bin:$PATH

export PATH

Note: Make sure to logout and login back. Verify that Java is found in the PATH by executing

$ which java
/opt/java/bin/java

$ java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

Macaw platform configuration is done through Macaw utility. The setup section documents the full details of the platform’s configuration steps along with details for each specific configuration item.

MACAW Setup

Macaw setup assists in generating the necessary platform and provisioner configuration. This is a mandatory step for the Macaw platform deployment. The section below provides the Macaw setup sequence with a detailed explanation on  various prompts.

Note: If using Macaw setup for the first, the user will be prompted for the EULA agreement. For most of the questions, the user can choose the default and make progress.

The below section shows the typical Macaw setup prompts with contextual help. Refer to additional documentation highlighted in blue (The highlighted text is not part of the Macaw setup output).

$ macaw setup

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@ The macaw setup will guide you through sequence of steps prompting you with default values and auto   @
@ generate platform and provisioner configuration. The default values are picked up from your existing  @
@ platform configuration (if any) or suggested based on your host configuration.                        @
@                                                                                                       @ 
@ If you are manually editing the platform configuration after auto generation, please refer to the     @
@ documentation.                                                                                        @
@ Location of Platform configuration file: ~/platform.cfg                                               @
@ Refer to ~/platform.README for more details on the platform configuration options.                    @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Please confirm to continue. [yes/no]: yes

+--------------------+----------------------------------------+
|  System            |  Details                               |
+--------------------+----------------------------------------+
|  OS                |  Linux                                 |
|  Kernel            |  3.10.0-327.36.2.el7.x86_64            |
|  Distribution      |  ('CentOS Linux', '7.2.1511', 'Core')  |
|  Memory Total      |  32774672 KB                           |
|  Memory Available  |  31925112 KB                           |
|  CPU Cores         |  8                                     |
+--------------------+----------------------------------------+

This is the FQDN of the platform host (your current host). FQDN needs to be fully DNS 
resolvable. Adding to /etc/hosts will not be sufficient. If you do not have DNS configured 
properly, you can provide IP address.

It is strongly suggested to use FQDN names. Since the services are containerized it is not 
going to work if you are simply adding the hostname and IP address mapping in the /etc/hosts file. 
The DNS resolution should be through a proper DNS server.
FQDN of platform Host [platform-190.qa.macaw.io]: platform-190.qa.macaw.io

Provide Service Hosts. If you have multiple hosts, please provide them comma separated.
DNS resolvable FQDN is recommended. Adding to /etc/hosts will not be sufficient. If you 
do not have DNS configured properly, you can provide IP address. 

The user needs to provide comma separated service hosts. Same recommendation on FQDN applies here as well.
This is the zookeeper endpoint. Default is chosen below. This assumes the user is running zookeeper 
on the platform host.

FQDN of services hosts (, separated if more than 1): platform-190.qa.macaw.io

This is the zookeeper endpoint. Default is chosen below. This assumes the user is running zookeeper 
on the platform host.

This is the user's zookeeper end point. Zookeeper can be run on any machine. By default macaw tool provisions the 
zookeeper container on the platform. Hence, the user sees the default selection as the platform host. If zookeeper is 
configured on a different host, provide the same host followed by the zookeeper port. 
Same recommendation on FQDN applies here as well.

Zookeeper endpoint [platform-190.qa.macaw.io:2181]: 

This is the kafka endpoint. Default is chosen below. This assumes the user is running kafka 
on the platform host.

This is the user's kafka end point. kafka can be run on any machine. By default macaw tool provisions the zookeeper container 
on the platform. Hence the user sees the default selection as the platform host. If kafka is configured on a different host, provide 
the same host followed by the kafka port. Same recommendation on FQDN applies here as well.

Kafka endpoint [platform-190.qa.macaw.io:9092]:  

This is the MYSQL DB endpoint. Default is chosen below. This assumes the user is running MYSQL  on the platform host. 

MYSQL endpoint [platform-190.qa.macaw.io:3306]:  

MYSQL User. If the user is already using an existing mysql installation, please make sure this user has the right access permissions. 
If the user is installing mysql via 'macaw infra install', this user will be automatically programmed for access. Also all default users of the system will be deleted. 

MYSQL Credentials. If the user is provisioning the mysql through macaw tool, the mysql is automatically provisioned  with this user. 
If not, this user is expected to be configured with read/write privileges to mysql and also enable  remote connection for the same user. 

MYSQL user [root]:  

MYSQL password for the above user.  

MYSQL password [Nl71dm31@12]:  

This is the cassandra DB endpoint. Default is chosen below. This assumes the user is running cassandra  on the platform host. 

Cassandra endpoint [platform-190.qa.macaw.io:9042]:  

Cassandra User. If the user is already using an existing cassandra installation, please make sure this user has the right access permissions. 
If the user is installing mysql via 'macaw infra install', this user will be automatically programmed for access. 
Also all default users of the system will be deleted. 

Cassandra DB credentials. cassandra/cassandra are the default credentials. If the user prefers to change this is the time to change.  
The user will be configured by the macaw tool in the cassandra DB. 

Cassandra user [cassandra]:  

Cassandra password for the above user. 

Cassandra password [cassandra]:  

This is the Elasticsearch DB endpoint. Default is chosen below. This assumes the user running Elasticsearch  on the platform host. 

Elasticsearch host (DB) [platform-190.qa.macaw.io:9300]:  

This is the Elasticsearch Log endpoint. Default is logging to elastic search is disabled. Please refer to documentation on how to enable this before bootstrapping the macaw platform. 

Elasticsearch host (Logging) []:  

MACAW platform uses common NFS mount across the platform and service VMs to be able to share  the data on the mount points. 
Please provide an end NFS mount point. By default this is pointing  to platform VM. 
If the user selects the default option, then the user needs to have the necessary NFS Server  packages installed on the platform VM. 
The below config would be reflected in the user's platform configuration. 
[nfs] host = <NFS Host> 
remote_mount = <Shared Mount on NFS> 
local_mount = <Local mount point to NFS mount>  

Note: If NFS utilities are not installed on the platform and Service VM, select the defaults and then skip  autosetup of the NFS mount 
and the user can update the  ~platform.cfg manually once they setup the NFS mounts manually by following the documentation. 

NFS Server [platform-190.qa.macaw.io]:  
NFS Server Mountpoint [/opt/macaw-shared]:  
Local NFS Mount [/opt/appd-shared-9ba77ea521865282]:  

This is deployment config directory where we store certificates and configurations. 
The sensitive data like certificates/keys are given restrictive permissions and can only be read & written by the current user. 
Certificate Repository:       $deploymentDir/certificates 
Common Configuration:         $deploymentDir/common 
Provisioner Configuration:    $deploymentDir/provisioner 
Systems Keys:                 $deploymentDir/keys  

Deployment config directory [/opt/macaw-config]:  

When self-signed certificates are generated, Macaw uses this as the pass phrase for the truststores. 
If the user is providing their own certificates (Please refer to documentation on how to) and if  the trust store is protected by 
passphrase please provide that passphrase here. 

Pass-phrase for certificate Trust stores [macaw@1234]:  

Platform configuration saved to: /home/macaw/platform.cfg 
Creating the configuration directory: /opt/macaw-config 
Service Provisioning Environment configuration saved to: /opt/macaw-config/provisioner/macaw-service-provisioner.properties 

Creating the platform macaw user 

Platform and Provisioner configuration are sync'ed. 

Key already exists at: /home/macaw/.ssh/id_rsa. 

Deploying SSH Key: /home/macaw/.ssh/id_rsa.pub to remote hosts 

SSH already deployed to: platform-190.qa.macaw.io for user: macaw 

Deploying self-signed certificates. Follow help/documentation for replacing them. 

Certificate Repository: /opt/macaw-config/certificates macaw-info:    

Certificate for: Self Signed CA macaw-info:    

Location: /opt/macaw-config/certificates/ca 

macaw-info:    Generating Self-Signed CA certificate 
macaw-info:    Importing Trusted CAs into Trust store 
macaw-info:    Certificate generation - Success 

macaw-info:    Certificate for: haproxy 
macaw-info:    Location: /opt/macaw-config/certificates/haproxy 
macaw-info:    Now generating the haproxy server certificate 
macaw-info:    Changing permissions and granting restrictive access 
macaw-info:    Certificate generation - Success 

macaw-info:    Certificate for: serviceregistry 
macaw-info:    Location: /opt/macaw-config/certificates/serviceregistry 
macaw-info:    Now generating the serviceregistry server certificate 
macaw-info:    Changing permissions and granting restrictive access 
macaw-info:    Certificate generation - Success 

macaw-info:    Certificate for: mdr 
macaw-info:    Location: /opt/macaw-config/certificates/mdr 
macaw-info:    Now generating the mdr server certificate 
macaw-info:    Changing permissions and granting restrictive access 
macaw-info:    Certificate generation - Success 

macaw-info:    Certificate for: dockerregistry 
macaw-info:    Location: /opt/macaw-config/certificates/dockerregistry 
macaw-info:    Now generating the dockerregistry server certificate 
macaw-info:    Changing permissions and granting restrictive access 
macaw-info:    Certificate generation - Success 
macaw-info:    Certificate for: kafka 

macaw-info:    Location: /opt/macaw-config/certificates/kafka 
macaw-info:    Now generating the kafka server certificate 
macaw-info:    Changing permissions and granting restrictive access 
macaw-info:    Certificate generation - Success 

Do you want NFS setup to be configured? This will require sudo access to the current user.  
Please confirm to continue. [yes/no]: yes 

Enabling NFS Server on the platform VM NFS Server enabled on Platform VM 
Verifying NFS mount... 

NFS mount successful: platform-190.qa.macaw.io 

Copying trust store to shared location for services: /opt/appd-shared-9ba77ea521865282/certificates/truststore 

MACAW Setup done. $ 

Once the Macaw setup is complete, the user should see platform.cfg, macaw-service-provisioner.properties and macaw-resource-profiles.json. If they need to make any modifications, they should be made before installing  infrastructure and platform configurations. For particular information on platform and provisioner configurations, the user can read the README files that reside in the same directory as configuration files.

Refer to the Configuration section on a precise explanation of platform and provisioner configuration. It further elaborates on what changes can be made and what needs to be done after changing the auto-generated configuration.

$ ls -lat ~/platform.cfg 
-rw-------. 1 macaw macaw 3478 Nov 25 12:11 /home/macaw/platform.cfg

$ ls -lat ~/platform.README 
-rw-r--r--. 1 macaw macaw 9298 Nov 25 12:11 /home/macaw/platform.README
$ 

$ ls -lat /opt/macaw-config/provisioner/macaw-service-provisioner.properties 
-rw-------. 1 macaw macaw 3152 Nov 25 12:11 /opt/macaw-config/provisioner/macaw-service-provisioner.properties

$ ls -lat /opt/macaw-config/provisioner/macaw-service-provisioner.README 
-rw-r--r--. 1 macaw macaw 6327 Nov 25 12:11 /opt/macaw-config/provisioner/macaw-service-provisioner.README

$ $ ls -lat /opt/macaw-config/provisioner/macaw-resource-profiles.json 
-rw-------. 1 cfx cfx 7955 Feb  4 14:24 /opt/macaw-config/provisioner/macaw-resource-profiles.json

Macaw setup auto-generates the platform configuration. Once the platform configuration is auto-generated, if you make any changes to the configuration, make sure to execute the command ‘macaw sync’ which keeps the platform configuration and provisioner configuration in sync.

To under details of the platform configuration, refer to the platform.README file under the home directory.

macaw-service-provisioner.properties is a JSON file which feeds Macaw service provisioner with environments where the user can provision their microservices. The JSON specification is an array of environment definitions. A default environment is mandated and created during the macaw setup with inputs provided for service hosts.

Each environment is required to provide this group of mandatory details:

• Fields of ID, name, type.

• Docker enabled Service End Hosts where your microservices are provisioned. Instead of providing service end hosts, the environment can also refer to a swarm cluster manager or Kubernetes master which in turn manages the cluster of docker nodes. (Refer to How to Enable Kubernetes Environment)

• Repositories: Array of MDR/Docker Registry pairs: Each repository is a pair of MDR (Service Meta Data Repository) and Docker Registry End point. During the provisioning time from the macaw-console, an option will be given to select a specific repository. Once a repository is selected, the user will be shown the available service blueprints from the MDR. When a provisioning request is issued after selecting a specific blueprint, the Macaw provisioner will be using the docker registry in that selected repo and will be downloading the container images. This data is absolutely necessary to be able to provision service blueprints via Macaw console. For more details on MDR, please refer to the Macaw documentation.

• List of capabilities. Below are the various capabilities an environment can provide.

restart-policy – This is the docker container restart policy setting. Any container deployed into this environment will inherit this policy.

https://docs.docker.com/engine/reference/run/#/restart-policies-restart

dns-configuration for containers – This is automatically appended to the capabilities during the macaw setup based on the user’s DNS settings defined in the platform configuration. The user can override this if needed.

log-configuration for containers – This setting controls the JSON log settings for the deployed containers into the environment.

volumes – These are the mandatory values that would be attached to every container deployed in this environment. For example, the user can store the truststore on a common NFS location and mount this to every container. The Service Blueprint doesn’t need to know about this specific mount point. These volumes are controlled and mandated by the environment.

storage – This defines the various mount/volumes that your environment can provide. Based on the name, the user’s service can request that it needs the specific volume. The provisioner/macaw console does the check before the provisioning if the user service blueprint is requesting volumes which are supported by the environment. If not, an error would be shown to the user. More details on the storage would be found under the Service Blueprints section.

env-variables – Mandatory environment variables for any container deployed in this environment.

resource-profiles – The list of resource profiles this environment can support. The resource profiles are defined in macaw-resource-profiles.json. Essentially through the resource profiles the user’s control the memory/cpu reservations/limits to the container. The environment enforces a default resource profile setting. If a service doesn’t request any specific resource profile the provisioner applies the default resource profile. For more details refer to macaw documentation. Sample JSON file.

opt-in-capabilities – This provides unique capabilities to the macaw platform to enable optional features like Debug, Memory Tracking for the deployed features. During the Service Blueprint deployment into a specific environment, the user can select any one or multiple of these optional capabilities to be applied to the deployed service. This JSON object defines each unique optional capability of the environment. Within the capability, you can specify resources like volumes, ports, labels, environment variables.  If the capability is selected during the provisioning time, then volumes, ports, labels, environments defined under this capability are applied to the deployed services. Using this unique way user can define custom optional services.

{
  "environments": [
    {
      "type": "standalone-docker",
      "id": "dda6e47c-e594-5a0a-9637-c747f090cc24",
      "machines": [
        {
          "pass-phrase": "",
          "ip": "macaw-s1.engr.cloudfabrix.com",
          "login": "macaw",
          "version": "7.1",
          "os": "centos"
        },
        {
          "pass-phrase": "",
          "ip": "macaw-s2.engr.cloudfabrix.com",
          "login": "macaw",
          "version": "7.1",
          "os": "centos"
        }
      ],
      "ui-pairs": [
        {
          "url": "https://macaw-p.engr.cloudfabrix.com",
          "name": "platform_uipair",
          "description": "Platform HAProxy and Tomcat",
          "volumes": [
            {
              "path": "/warfiles",
              "sub-path": "platform_uipair",
              "name": "MACAW_PLATFORM_MOUNT",
              "read-write-mode": "rw"
            }
          ],
          "port": 443
        }
      ],
      "name": "macaw-default-env-01",
      "repositories": [
        {
          "mdr": {
            "protocol": "https",
            "name": "Internal MDR",
            "host": "10.95.100.10",
            "repo": "dev",
            "token": "6f009a42-9a9a-4cfa-a248-fb82deff8a12",
            "version": "v2",
            "port": 8639,
            "description": "This is the internal development MDR"
          },
          "docker-registry": {
            "username": "macaw",
            "protocol": "https",
            "name": "Internal Docker Registry",
            "port": 5000,
            "host": "cfx-docker-01.engr.cloudfabrix.com",
            "password": "password",
            "email": "macaw@www.macaw.io",
            "description": "This is the internal development docker registry"
          },
          "name": "cfx-internal-repo",
          "description": "Development MDR and Docker Registry Repository"
        },
        {
          "mdr": {
            "protocol": "http",
            "name": "macaw onprem MDR",
            "token": "dda6e47c-e594-5a0a-9637-c747f090cc24",
            "repo": "dev",
            "host": "macaw-p.engr.cloudfabrix.com",
            "version": "v2",
            "port": 8637,
            "description": "This is the onprem MDR"
          },
          "docker-registry": {
            "username": "macaw",
            "protocol": "https",
            "name": "macaw onprem Docker Registry",
            "port": 5000,
            "host": "macaw-p.engr.cloudfabrix.com",
            "password": "macaw@local",
            "email": "macaw@local.com",
            "description": "This is the onprem docker registry"
          },
          "name": "onprem MDR/Docker",
          "description": "onprem installed MDR/Docker"
        }
      ],
      "capabilities": {
        "restart-policy": {
          "name": "unless-stopped"
        },
        "installation": [],
        "resource-profiles": {
          "default": "macaw-rp-mwr.small",
          "supported": [
            "macaw-rp-mwr.small",
            "macaw-rp-mwr.medium",
            "macaw-rp-mwr.large",
            "macaw-rp-mwr.2xlarge",
            "macaw-rp-mwr.4xlarge",
            "macaw-rp-mwr.8xlarge",
            "macaw-rp-mwr.iota",
            "macaw-rp-mwr.iota-small",
            "macaw-rp-mwr.iota-medium"
          ]
        },
        "env-variables": [
          {
            "name": "MACAW_SSL_TRUSTSTORE_LOCATION",
            "value": "/opt/macaw/secrets/truststore/ca_truststore"
          },
          {
            "name": "MACAW_SSL_TRUSTSTORE_PASSWORD",
            "value": "macaw@1234"
          }
        ],
        "labels": [
          {
            "name": "io.macaw.type",
            "value": "macaw-micro-services"
          }
        ],
        "storage": [
          {
            "path": "/opt/java",
            "name": "JAVA_1.8",
            "read-write-mode": "ro"
          },
          {
            "path": "/opt/macaw-platform-dda6e47ce5945a0a",
            "name": "MACAW_PLATFORM_MOUNT",
            "read-write-mode": "rw"
          },
          {
            "path": "/opt/appd-shared-dda6e47ce5945a0a",
            "name": "MACAW_ENVIRONMENT_DEFAULT_MOUNT",
            "read-write-mode": "rw"
          }
        ],
        "dns-configuration": {
          "search-domain": [],
          "opts": [],
          "server": []
        },
        "volumes": [
          {
            "path": "/opt/java",
            "name": "JAVA_1.8",
            "read-write-mode": "ro"
          },
          {
            "path": "/opt/macaw/secrets/truststore",
            "sub-path": "certificates/truststore",
            "name": "MACAW_PLATFORM_MOUNT",
            "read-write-mode": "ro"
          }
        ],
        "log-configuration": {
          "enable": true,
          "driver": {
            "name": "json-file",
            "opts": {
              "max-size": "10m",
              "max-file": 5
            }
          }
        }
      },
      "opt-in-capabilities": {
        "debug": {
          "name": "Macaw Debug",
          "labels": [
            {
              "name": "io.macaw.debug.port",
              "value": "enabled"
            }
          ],
          "env-variables": [
            {
              "name": "MACAW_ENABLE_DEBUG_PORT",
              "value": "true"
            }
          ],
          "actions": [],
          "volumes": [],
          "ports": [
            {
              "protocol": "tcp",
              "endpoint": "Debug",
              "type": "dynamic",
              "name": "Macaw Debug Port",
              "port": 8787
            }
          ],
          "description": "Enables debug port for Macaw Microservices"
        },
        "nmt-summary": {
          "name": "Native Memory Tracking Summary",
          "labels": [
            {
              "name": "io.macaw.jvm.nativememory",
              "value": "summary"
            }
          ],
          "env-variables": [
            {
              "name": "MACAW_JVM_ARGS_ADDITIONAL",
              "value": "-XX:NativeMemoryTracking=summary"
            }
          ],
          "actions": [],
          "volumes": [],
          "ports": [],
          "description": "Enables Native Memory Tracking Summary flag on the JVM for Macaw Microservices"
        },
        "logtoES": {
          "name": "Elasticsearch Logging",
          "labels": [
            {
              "name": "io.macaw.logging.elasticsearch",
              "value": "true"
            }
          ],
          "env-variables": [
            {
              "name": "MACAW_PLATFORM_LOGGING_URL",
              "value": "http://macaw-p.engr.cloudfabrix.com:9200/_bulk"
            }
          ],
          "actions": [],
          "volumes": [],
          "ports": [],
          "description": "Enables Elasticsearch Logging for Macaw Microservice"
        },
        "nmt-detail": {
          "name": "Native Memory Tracking Detail",
          "labels": [
            {
              "name": "io.macaw.jvm.nativememory",
              "value": "detail"
            }
          ],
          "env-variables": [
            {
              "name": "MACAW_JVM_ARGS_ADDITIONAL",
              "value": "-XX:NativeMemoryTracking=detail"
            }
          ],
          "actions": [],
          "volumes": [],
          "ports": [],
          "description": "Enables Native Memory Tracking Detail flag on the JVM for Macaw Microservices"
        },
        "adpm": {
          "name": "Macaw APM with ADPM Provider",
          "labels": [
            {
              "name": "io.macaw.apm.provider",
              "value": "adpm"
            }
          ],
          "env-variables": [
            {
              "name": "MACAW_APM_PROVIDER",
              "value": "adpm"
            },
            {
              "name": "MACAW_APM_CENTRAL_REPO",
              "value": "macaw-p.engr.cloudfabrix.com:8181"
            }
          ],
          "actions": [
            {
              "url": "https://demo.macaw.io/adpm",
              "name": "Launch ADPM"
            }
          ],
          "volumes": [
            {
              "path": "/opt/macaw/apm",
              "sub-path": "apm/adpm/agent",
              "name": "MACAW_PLATFORM_MOUNT",
              "read-write-mode": "rw"
            }
          ],
          "ports": [],
          "description": "Enables APM capabilities for Macaw Microservices"
        }
      }
    }
  ]
}

Macaw Platform segregates compute resources into environments. During the Service Blueprint deployment, you provision into an environment which is either backed by individual docker hosts or a Kubernetes cluster or swarm cluster.

Macaw supports the following provisioning environments:

1. Standalone Docker Hosts
2. Swarm Cluster
3. Kubernetes Cluster (Kubernetes v1.5.2)

A group of individual docker hosts can be grouped together under an environment. The only criteria for grouping the nodes is: the storage definition defined under the environment should be accessible from all the nodes in the environment. For example, if you have defined a storage mount like below in the environment, then it is expected that all the nodes provide this storage.

 "storage": [
 {
 "path": "/opt/java",
 "name": "JAVA_1.8",
 "read-write-mode": "ro"
 }
 ]

Though Macaw Platform doesn’t enforce this, it is highly recommended to group hosts with same Docker Version, Storage Mounts, CPU/Memory resources into an environment.

As part of the Macaw setup, it is mandated to provide a single service host which is then used to create a default environment.

Below is how compute resources are provided in the environment definition.

  "machines": [
 {
 "pass-phrase": "",
 "ip": "macaw-s1.engr.cloudfabrix.com",
 "login": "macaw",
 "version": "7.1",
 "os": "centos"
 },
 {
 "pass-phrase": "",
 "ip": "macaw-s2.engr.cloudfabrix.com",
 "login": "macaw",
 "version": "7.1",
 "os": "centos"
 }
 ]

During the Macaw setup phase, password less SSH communication is established to these service endpoints from the platform host. Without this password-less SSH provisioning requests would fail.

Macaw Platform provides the capability to provision services onto the Kubernetes cluster. Below are the requirements before a Kubernetes environment can be created.

1. Kubernetes Master
2. Credentials – User/password or Token based Authentication
3. Namespace – If no namespace is available, default can be provided. Any non-default namespace should be created upfront on the Kubernetes.
4. Mandatory PVCs and any optional PVCs required by services should be created up front on the K8 cluster

In the below section we will cover in detail on how to enable Kubernetes environment in Macaw platform. Below is a sample K8 environment JSON spec.

{
  "environments": [
    {
      "id": "9ba77ea5-2186-5282-9b88-93b373a59f32",
      "name": "macaw-k8-env-01",
      "type": "kubernetes",
      "kubernetes-master": {
        "url": "http://k8-master-qa.qa.engr.cloudfabrix.com:8080",
        "username": "user@k8",
        "password": "password",
        "namespace": "macaw",
        "auth": "basic-auth"
      },
      "repositories": [
        {
          "name": "macaw-k8-repo",
          "description": "Internal MDR and Docker Registry Repository",
          "mdr": {
            "protocol": "https",
            "name": "Internal MDR",
            "host": "1.1.1.1",
            "repo": "dev",
            "token": "6f009a42-9a9a-4cfa-a248-fb82d12f8a01",
            "version": "v2",
            "port": 8639,
            "description": "This is the internal development MDR"
          },
          "docker-registry": {
            "username": "foo",
            "protocol": "https",
            "name": "Internal Docker Registry",
            "port": 5000,
            "host": "docker-01.local.host",
            "password": "123456",
            "email": "macaw@www.macaw.io",
            "description": "This is the internal development docker registry"
          }
        }
      ],
      "ui-pairs": [
        {
          "url": "https://macaw-p.engr.cloudfabrix.com",
          "name": "platform_uipair",
          "description": "Platform HAProxy and Tomcat",
          "volumes": [
            {
              "path": "/warfiles",
              "sub-path": "platform_uipair",
              "name": "MACAW_PLATFORM_MOUNT",
              "read-write-mode": "rw"
            }
          ],
          "port": 443
        }
      ],
      "capabilities": {
        "restart-policy": {
          "name": "unless-stopped"
        },
        "installation": [],
        "resource-profiles": {
          "default": "macaw-rp-mwr.small",
          "supported": [
            "macaw-rp-mwr.small",
            "macaw-rp-mwr.medium",
            "macaw-rp-mwr.large",
            "macaw-rp-mwr.2xlarge",
            "macaw-rp-mwr.4xlarge",
            "macaw-rp-mwr.8xlarge",
            "macaw-rp-mwr.iota",
            "macaw-rp-mwr.iota-small",
            "macaw-rp-mwr.iota-medium"
          ]
        },
        "env-variables": [
          {
            "name": "MACAW_ENV_VARIABLE_NAMES_TRANSLATOR",
            "value": "FIXED_SET_ENV_NAMES_TRANSLATOR"
          },
          {
            "name": "MACAW_SSL_TRUSTSTORE_LOCATION",
            "value": "/opt/macaw/secrets/truststore/ca_truststore"
          },
          {
            "name": "MACAW_SSL_TRUSTSTORE_PASSWORD",
            "value": "macaw@1234"
          }
        ],
        "labels": [
          {
            "name": "io.macaw.type",
            "value": "macaw-micro-services"
          }
        ],
        "storage": [
          {
            "persistent-volume-claim": {
              "claim-name": "macaw.installation.java",
              "claim-description": "Java Host Volume"
            },
            "name": "JAVA_1.8",
            "read-write-mode": "ro"
          },
          {
            "persistent-volume-claim": {
              "claim-name": "macaw.platform",
              "claim-description": "K8 Macaw Platform PVC Claim."
            },
            "name": "MACAW_PLATFORM_MOUNT",
            "read-write-mode": "rw"
          },
          {
            "persistent-volume-claim": {
              "claim-name": "macaw.environment.default",
              "claim-description": "K8 Macaw Environment Default PVC"
            },
            "name": "MACAW_ENVIRONMENT_DEFAULT_MOUNT",
            "read-write-mode": "rw"
          }
        ],
        "dns-configuration": {
          "search-domain": [],
          "opts": [],
          "server": []
        },
        "volumes": [
          {
            "path": "/opt/java",
            "name": "JAVA_1.8",
            "read-write-mode": "ro"
          },
          {
            "path": "/opt/macaw/secrets/truststore",
            "sub-path": "certificates/truststore",
            "name": "MACAW_PLATFORM_MOUNT",
            "read-write-mode": "ro"
          }
        ],
        "log-configuration": {
          "enable": true,
          "driver": {
            "name": "json-file",
            "opts": {
              "max-size": "10m",
              "max-file": 5
            }
          }
        }
      },
      "opt-in-capabilities": {
        "debug": {
          "name": "Macaw Debug",
          "labels": [
            {
              "name": "io.macaw.debug.port",
              "value": "enabled"
            }
          ],
          "env-variables": [
            {
              "name": "MACAW_ENABLE_DEBUG_PORT",
              "value": "true"
            }
          ],
          "actions": [],
          "volumes": [],
          "ports": [
            {
              "protocol": "tcp",
              "endpoint": "Debug",
              "type": "dynamic",
              "name": "Macaw Debug Port",
              "port": 8787
            }
          ],
          "description": "Enables debug port for Macaw Microservices"
        },
        "nmt-summary": {
          "name": "Native Memory Tracking Summary",
          "labels": [
            {
              "name": "io.macaw.jvm.nativememory",
              "value": "summary"
            }
          ],
          "env-variables": [
            {
              "name": "MACAW_JVM_ARGS_ADDITIONAL",
              "value": "-XX:NativeMemoryTracking=summary"
            }
          ],
          "actions": [],
          "volumes": [],
          "ports": [],
          "description": "Enables Native Memory Tracking Summary flag on the JVM for Macaw Microservices"
        },
        "logtoES": {
          "name": "Elasticsearch Logging",
          "labels": [
            {
              "name": "io.macaw.logging.elasticsearch",
              "value": "true"
            }
          ],
          "env-variables": [
            {
              "name": "MACAW_PLATFORM_LOGGING_URL",
              "value": "http://macaw-p.engr.cloudfabrix.com:9200/_bulk"
            }
          ],
          "actions": [],
          "volumes": [],
          "ports": [],
          "description": "Enables Elasticsearch Logging for Macaw Microservice"
        },
        "nmt-detail": {
          "name": "Native Memory Tracking Detail",
          "labels": [
            {
              "name": "io.macaw.jvm.nativememory",
              "value": "detail"
            }
          ],
          "env-variables": [
            {
              "name": "MACAW_JVM_ARGS_ADDITIONAL",
              "value": "-XX:NativeMemoryTracking=detail"
            }
          ],
          "actions": [],
          "volumes": [],
          "ports": [],
          "description": "Enables Native Memory Tracking Detail flag on the JVM for Macaw Microservices"
        },
        "adpm": {
          "name": "Macaw APM with ADPM Provider",
          "labels": [
            {
              "name": "io.macaw.apm.provider",
              "value": "adpm"
            }
          ],
          "env-variables": [
            {
              "name": "MACAW_APM_PROVIDER",
              "value": "adpm"
            },
            {
              "name": "MACAW_APM_CENTRAL_REPO",
              "value": "macaw-p.engr.cloudfabrix.com:8181"
            }
          ],
          "actions": [
            {
              "url": "https://demo.macaw.io/adpm",
              "name": "Launch ADPM"
            }
          ],
          "volumes": [
            {
              "path": "/opt/macaw/apm",
              "sub-path": "apm/adpm/agent",
              "name": "MACAW_PLATFORM_MOUNT",
              "read-write-mode": "rw"
            }
          ],
          "ports": [],
          "description": "Enables APM capabilities for Macaw Microservices"
        }
      }
    }
  ]
}

• Kubernetes Master Endpoint

Macaw Platform communicates with the Kubernetes Master for provisioning/de-provisioning/scale up/scale down/rolling update of Service Pods. Macaw supports simple basic-auth or token based authentication to the Kubernetes cluster.

Simple Basic Authentication

"kubernetes-master": {
        "url": "http://k8-master-qa.qa.engr.cloudfabrix.com:8080",
        "username": "user@k8",
        "password": "password",
        "namespace": "macaw",
        "auth": "basic-auth"
      }

Token Authentication

"kubernetes-master": {
        "url": "http://k8-master-qa.qa.engr.cloudfabrix.com:8080",
        "username": "user@k8",
        "password": "password",
        "namespace": "macaw",
        "auth": "basic-auth"
      }

Namespaces are virtual clusters in Kubernetes. Macaw Platform provides the ability to attach a namespace to the environment. The only restriction is the namespace should be created on the cluster and the storage PVCs belong to this namespace.

• Storage PVCs

This is the most important section. Below is how storage mount points defined for a Kubernetes environment.

        "storage": [
          {
            "persistent-volume-claim": {
              "claim-name": "macaw.installation.java",
              "claim-description": "Java Host Volume"
            },
            "name": "JAVA_1.8",
            "read-write-mode": "ro"
          },
          {
            "persistent-volume-claim": {
              "claim-name": "macaw.platform",
              "claim-description": "K8 Macaw Platform PVC Claim."
            },
            "name": "MACAW_PLATFORM_MOUNT",
            "read-write-mode": "rw"
          },
          {
            "persistent-volume-claim": {
              "claim-name": "macaw.environment.default",
              "claim-description": "K8 Macaw Environment Default PVC"
            },
            "name": "MACAW_ENVIRONMENT_DEFAULT_MOUNT",
            "read-write-mode": "rw"
          }
        ]

In the above section when we consider the JAVA_1.8 storage mount, shown below is how it has to be interpreted.

Macaw Platform refers to this storage definition by name JAVA_1.8 and this refers to the PVC ‘macaw.installation.java’ in the Kubernetes.  It is mandatory for this PVC to be defined and created upfront on the Kubernetes master.

It is mandatory for the storage mounts/PVCs below to be defined in the environment as these are utilized by Macaw platform.

1. JAVA_1.8 (Name can be different) storage mount and associated PVC to be created. This storage mount provides the Java installation path on the host. Therefore it is mandatory for all the Kubernetes hosts to have this mandatory Java volume. This can be avoided, if you are building Microservices with Java embedded into the container.
2. MACAW_PLATFORM_MOUNT (Name can be different) storage mount and associated PVC. This is an NFS mount point pointing to the shared NFS mount from the platform host. Platform Host shares the necessary self signed certificate truststores via the NFS and this has to be mounted to all the containers.

Note: Make sure you edit the /etc/exportfs on the platform host and add the platform directory export entries to all the Kubernetes nodes. Once this is done, make sure to refresh export definitions via export ‘sudo exportfs -avr’

more /etc/exports

/opt/macaw-shared/platform k8-m1(rw,async,no_root_squash)
/opt/macaw-shared/platform k8-m2(rw,async,no_root_squash)
/opt/macaw-shared/platform k8-m3(rw,async,no_root_squash)
/opt/macaw-shared/platform k8-m4(rw,async,no_root_squash)

Below is a sample Kubernetes JSON spec to create the necessary Physical Volumes and PVCs for the above mandatory storage mounts.

{
  "apiVersion": "v1",
  "kind": "List",
  "items": [
    {
      "apiVersion": "v1",
      "kind": "PersistentVolume",
      "metadata": {
        "labels": {
          "macaw_purpose": "platform"
        },
        "name": "macaw.platform"
      },
      "spec": {
        "accessModes": [
          "ReadWriteMany"
        ],
        "capacity": {
          "storage": "20Gi"
        },
        "nfs": {
          "path": "/opt/macaw-shared/platform",
          "server": "10.95.110.100"
        },
        "persistentVolumeReclaimPolicy": "Retain"
      }
    }
    {
      "apiVersion": "v1",
      "kind": "PersistentVolume",
      "metadata": {
        "labels": {
          "macaw_purpose": "java"
        },
        "name": "macaw.java"
      },
      "spec": {
        "accessModes": [
          "ReadOnlyMany"
        ],
        "capacity": {
          "storage": "5Gi"
        },
        "hostPath": {
          "path": "/opt/java"
        },
        "persistentVolumeReclaimPolicy": "Retain"
      }
    },
    {
      "apiVersion": "v1",
      "kind": "PersistentVolumeClaim",
      "metadata": {
        "name": "macaw.platform",
        "namespace": "demomacawio"
      },
      "spec": {
        "accessModes": [
          "ReadWriteMany"
        ],
        "resources": {
          "requests": {
            "storage": "20Gi"
          }
        },
        "selector": {
          "matchLabels": {
            "macaw_purpose": "platform"
          }
        }
      }
    }
    {
      "apiVersion": "v1",
      "kind": "PersistentVolumeClaim",
      "metadata": {
        "name": "macaw.installation.java",
        "namespace": "demomacawio"
      },
      "spec": {
        "accessModes": [
          "ReadOnlyMany"
        ],
        "resources": {
          "requests": {
            "storage": "5Gi"
          }
        },
        "selector": {
          "matchLabels": {
            "macaw_purpose": "java"
          }
        }
      }
    }
  ]
}

Now execute the kubectl CLI on the Kubernetes master and create these resources like below. The above JSON spec is stored in the file storage.json.

kubectl create -f storage.json

Now verify the storage creation using the below commands

kubectl get pv
kubectl get pvc --namespace=demomacawio

Macaw platform installation involves the installation of Macaw infrastructure and platform components. The below sequence of commands will install the Macaw platform. The prerequisite for platform installation is the Macaw setup, completed in the previous step.

Various Installation Commands

macaw infra install --tag <tag>
macaw apm install --tag <tag>
macaw platform dbinit --tag <tag>
macaw platform install --tag <tag>
macaw tools install --tag <tag> --service macaw-mdr
macaw tools install --tag 2.3.1 --service docker-registry

Stated here are additional details on infrastructure, apm, platform, and tools installation sections of the Macaw platform.

Infrastructure Installation

As part of the Macaw infrastructure installation, the following components would be provisioned to present critical services to the Macaw platform. This installation is mandatory for the Macaw Platform.

Zookeeper
Kafka
Elasticsearch
Mysql
Cassandra
HAProxy
Tomcat
Redis

Install infrastructure using the below command.

$ macaw infra install --tag macaw-v0.9.4
		***********************************************
		**************** Macaw Infra  *****************
		***********************************************


Bootstrapping zookeeper with tag: macaw-v0.9.4
WARNING: Updating/Saving docker credentials to : /home/macaw/.docker/macawauth.json
...
Bootstrapping zookeeper: SUCCESS

Bootstrapping kafka with tag: macaw-v0.9.4
...
Bootstrapping kafka: SUCCESS

Bootstrapping elasticsearch with tag: macaw-v0.9.4
...
Bootstrapping elasticsearch: SUCCESS

Bootstrapping cassandra with tag: macaw-v0.9.4
...
Bootstrapping cassandra: SUCCESS

Bootstrapping mysql with tag: macaw-v0.9.4
...
Bootstrapping mysql: SUCCESS

Bootstrapping haproxy with tag: macaw-v0.9.4
...
Bootstrapping haproxy: SUCCESS

Bootstrapping macaw-tomcat with tag: macaw-v0.9.4
...
Bootstrapping macaw-tomcat: SUCCESS

Bootstrapping redis with tag: macaw-v0.9.4
...
Bootstrapping redis: SUCCESS

Done Bootstrapping the Macaw Infra.
Initialization Done

Status of the Macaw Infra:
+-----------------+------------------------+----------------+----------------+-------------------------------------------+-----------+------------+
|  Name           |  Status                |  Container Id  |  Image Tag     |  Container Name                           |  Mem(MB)  |  Version   |
+-----------------+------------------------+----------------+----------------+-------------------------------------------+-----------+------------+
|  zookeeper      |  Up 35 seconds         |  936f2c2f8b8d  |  macaw-v0.9.4  |  zookeeper_39426d8f-dc83-4d8f-8980-44843  |  60       |  0.10.2.0  |
|  kafka          |  Up 24 seconds         |  d44429853eb2  |  macaw-v0.9.4  |  kafka_8fb42cfe-5936-4918-bc04-d201e0453  |  263      |  0.10.2.0  |
|  elasticsearch  |  Up 19 seconds         |  012e5bbdb2a9  |  macaw-v0.9.4  |  elasticsearch_bf79b13a-e75f-46ab-a5ae-8  |  310      |  1.7.5     |
|  cassandra      |  Up 17 seconds         |  2ab48ca422ee  |  macaw-v0.9.4  |  cassandra_b82a3ae1-f4c7-4f2e-b269-07a61  |  1251     |  2.2.6     |
|  mysql          |  Up 7 seconds          |  582c5ac342f1  |  macaw-v0.9.4  |  mysql_0c391328-cf97-426f-82f2-527de8095  |  186      |  5.5.49    |
|  haproxy        |  Up 1 second           |  b947e443d000  |  macaw-v0.9.4  |  haproxy_254cfc7f-2ed2-4ec0-a40d-9015292  |  97       |  1.7.1     |
|  macaw-tomcat   |  Up Less than a secon  |  7b80495e68d7  |  macaw-v0.9.4  |  macaw-tomcat_16daf0b5-d6b9-49cd-91d8-ba  |  60       |  8.0.37    |
|  redis          |  Up Less than a secon  |  5a868d71ae66  |  macaw-v0.9.4  |  redis_05032cff-e552-4268-918a-78441f2c9  |  12       |  3.2.8     |
|  NFS Server     |  [OK]                  |  --            |  --            |  --                                       |  --       |  v4        |
|  docker         |  [OK]                  |  --            |  --            |  --                                       |  --       |  1.13.1    |
+-----------------+------------------------+----------------+----------------+-------------------------------------------+-----------+------------+

APM Installation

APM Install commands would install the Performance Monitoring Agent and Collector. This is an optional installation.

macaw apm install --tag macaw-v0.9.4
		***********************************************
		************* Macaw APM Services **************
		***********************************************


Bootstrapping macaw-apm-agent with tag: macaw-v0.9.4
WARNING: Updating/Saving docker credentials to : /home/macaw/.docker/macawauth.json
...................................................................
Bootstrapping macaw-apm-agent: SUCCESS

Bootstrapping macaw-apm-collector with tag: macaw-v0.9.4
...................................................................
Bootstrapping macaw-apm-collector: SUCCESS

Status of the Macaw APM:
+-----------------------+------------------------+----------------+----------------+---------------------------------+-----------+-------------+
|  Name                 |  Status                |  Container Id  |  Image Tag     |  Container Name                 |  Mem(MB)  |  Version    |
+-----------------------+------------------------+----------------+----------------+---------------------------------+-----------+-------------+
|  macaw-apm-agent      |  Up 4 seconds          |  fd701a7e5bbe  |  macaw-v0.9.4  |  macaw-apm-agent_26b6a333-4153  |  1        |  0.9.16.14  |
|  macaw-apm-collector  |  Up Less than a secon  |  7ae28569b33b  |  macaw-v0.9.4  |  macaw-apm-collector_63fe6707-  |  37       |  0.9.16.14  |
+-----------------------+------------------------+----------------+----------------+---------------------------------+-----------+-------------+

ADPM UI can be accessed at - https://192.168.33.10/adpm

Platform DBInit

Platform dbinit command initializes the database sections of mysql and Cassandra. This is mandatory step that needs to be executed before proceeding with the installation of platform core components.

macaw platform dbinit --tag macaw-v0.9.4
 
*******WARNING**********WARNING********WARNING******** 
 
This will initialize the DB schema for macaw platform services. Initialization involves dropping existing 
tables/data and creating new. Any data for the macaw platform services in the DB will be lost.

WARNING: The dbinit need to be done only once before the platform is installed. Doing this initialization post
platform installation or when platform is active/runing, will result in data inconsistencies and loss of data.
 

Do you want to continue with DB Initialization? [yes/no]: yes

Bootstrapping macaw-dbinit with tag: macaw-v0.9.4
Note: macaw-dbinit is a transient service and will be removed.
WARNING: Updating/Saving docker credentials to : /home/macaw/.docker/macawauth.json
...
Bootstrapping macaw-dbinit: SUCCESS

Verifying checksum of the DB Init package

Platform Admin User [admin@www.macaw.io]: 
Password :
Re-enter Password :


2017-06-21 14:49:45 : macaw-dbinit-info: DB Init - Start
2017-06-21 14:49:45 : macaw-dbinit-info: MYSQL Endpoint: 192.168.33.10:3306
2017-06-21 14:49:45 : macaw-dbinit-info: Cassandra Endpoint: 192.168.33.10:9042
2017-06-21 14:49:45 : macaw-dbinit-info: Initializing DB Endpoints
2017-06-21 14:49:45 : macaw-dbinit-info: Service Registry Init - Start
2017-06-21 14:49:46 : macaw-dbinit-info: Service Registry Init - Done
2017-06-21 14:49:47 : macaw-dbinit-info: Services Init - Start
2017-06-21 14:49:57 : macaw-dbinit-info: Programming Locker master key - Start
2017-06-21 14:50:15 : macaw-dbinit-info: Programming Locker master key - Done
2017-06-21 14:50:15 : macaw-dbinit-info: Services Init - Done
2017-06-21 14:50:15 : macaw-dbinit-info: DB Init - Done

macaw-db-init: DB Init Successful
macaw-db-init: Cleaning up temporary files...
macaw-db-init: Done.

Platform Installation

As part of the platform installation, the following components will be provisioned.

Service Registry
Notification Manager
Service Provisioner
Identity Service
User Preferences
Console UI

Install the platform using the below command.

macaw platform install --tag macaw-v0.9.4
		***********************************************
		************** Macaw Platform  ****************
		***********************************************


Bootstrapping service-registry with tag: macaw-v0.9.4
WARNING: Updating/Saving docker credentials to : /home/macaw/.docker/macawauth.json
...
Bootstrapping service-registry: SUCCESS

Bootstrapping notification-manager with tag: macaw-v0.9.4
...
Bootstrapping notification-manager: SUCCESS

Bootstrapping identity with tag: macaw-v0.9.4
...
Bootstrapping identity: SUCCESS

Bootstrapping service-provisioner with tag: macaw-v0.9.4
...
Bootstrapping service-provisioner: SUCCESS

Bootstrapping user-preferences with tag: macaw-v0.9.4
...
Bootstrapping user-preferences: SUCCESS

Bootstrapping console-ui with tag: macaw-v0.9.4
...
Bootstrapping console-ui: SUCCESS

Status of the Macaw Platform:
+------------------------+-----------------+----------------+----------------+---------------------------------+-----------+-----------+
|  Name                  |  Status         |  Container Id  |  Image Tag     |  Container Name                 |  Mem(MB)  |  Version  |
+------------------------+-----------------+----------------+----------------+---------------------------------+-----------+-----------+
|  service-registry      |  Up 45 seconds  |  a97ffb9b39c3  |  macaw-v0.9.4  |  service-registry_2a0b5d28-eca  |  298      |  0.9.4    |
|  notification-manager  |  Up 24 seconds  |  864342b4eadf  |  macaw-v0.9.4  |  notification-manager_9a980bea  |  151      |  0.9.4    |
|  identity              |  Up 13 seconds  |  f834e9b9e74f  |  macaw-v0.9.4  |  identity_4efca2dd-ac25-4d2b-9  |  209      |  0.9.4    |
|  service-provisioner   |  Up 6 seconds   |  63dab143112d  |  macaw-v0.9.4  |  service-provisioner_9f815ed6-  |  110      |  0.9.4    |
|  user-preferences      |  Up 5 seconds   |  7118dc077872  |  macaw-v0.9.4  |  user-preferences_53bbf6ea-f3e  |  73       |  0.9.4    |
|  console-ui            |  Up 2 seconds   |  ce48a2bec347  |  macaw-v0.9.4  |  console-ui_6d8aa533-17aa-44a1  |  1        |  0.9.4    |
+------------------------+-----------------+----------------+----------------+---------------------------------+-----------+-----------+

Console UI can be accessed at - https://192.168.33.10

MDR/Docker Tools Installation

As part of the tools installation, following components would be provisioned.

Macaw MDR
Docker Registry
Note: Docker Registry is pulled from Docker Public Registry and use 2.3.1 version.

Install tools using the below command.

$ macaw tools install --tag macaw-v0.9.4 --service macaw-mdr
<Please get the tag from the documentation for the current macaw platform version.>
        ***********************************************
        **************** MACAW Tools  *****************
        ***********************************************


Bootstrapping macaw-mdr with tag: macaw-v0.9.4
macaw-info: Creating directory and granting restrictive permissions : /opt/macaw-tools/mdr/auth
macaw-info: Creating directory and granting restrictive permissions : /opt/macaw-tools/mdr/repository
WARNING: Updating/Saving docker credentials to : /home/macaw/.docker/macawauth.json
................................................................................
................................................................................
................................................................................
...............
Bootstrapping macaw-mdr: SUCCESS

Configuration details at: /opt/macaw-config/macaw-tools/provisionerconfig.json

Configuration details at: /opt/macaw-config/macaw-tools/macawpublish.globals

Done Bootstrapping the Macaw Tools.

$ 
$ macaw tools install --tag 2.3.1 --service docker-registry
<Please get the tag from the documentation for the current macaw platform version.>
        ***********************************************
        **************** MACAW Tools  *****************
        ***********************************************

Bootstrapping docker-registry with tag: 2.3.1
macaw-info: Creating directory and granting restrictive permissions : /opt/macaw-tools/dockerregistry

NOTE: Downloading the docker registry from the docker public repository.
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
...
Bootstrapping docker-registry: SUCCESS

Docker Registry is installed with self signed private certificate. 
Below steps/configuration are needed for the any docker host to be able to talk to this registry.

Step 1: Login to the docker Host(s) and execute the below 

    sudo mkdir -p /etc/docker/certs.d/192.168.33.10:5000

Step 2: Download ca.crt from the platform host to the docker host(s). 

    sudo scp macaw@192.168.33.10:/opt/macaw-config/certificates/ca/ca.crt /etc/docker/certs.d/192.168.33.10:5000

Step 3: Verify docker login

    docker login -u macaw -p macaw@local -e macaw@local.com 192.168.33.10:5000

Configuration details at: /opt/macaw-config/macaw-tools/dockeregistry.txt

Configuration details at: /opt/macaw-config/macaw-tools/provisionerconfig.json

Configuration details at: /opt/macaw-config/macaw-tools/macawpublish.globals

Done Bootstrapping the Macaw Tools.

$ macaw tools status

Status of the Macaw Tools:
+-------------------+----------------+----------------+----------------+-------------------------------------------+-----------+-----------+
|  Name             |  Status        |  Container Id  |  Image Tag     |  Container Name                           |  Mem(MB)  |  Version  |
+-------------------+----------------+----------------+----------------+-------------------------------------------+-----------+-----------+
|  macaw-mdr        |  Up 4 minutes  |  68e7ccd64ba6  |  macaw-v0.9.4  |  macaw-mdr_6ee9f449-aa86-4b4b-8218-7e86d  |  10       |  --       |
|  docker-registry  |  Up 3 minutes  |  60f259751b74  |  2.3.1         |  docker-registry_90f75822-8ff7-4bb2-9af1  |  16       |  --       |
+-------------------+----------------+----------------+----------------+-------------------------------------------+-----------+-----------+
$ 

Once the SDK is downloaded, extract the file by using the following commands based on the file formats

unzip macaw-sdk-version.zip (or) tar xvzf macaw-sdk-version.tar.gz

Macaw platform supports various programming languages that the developer can choose to develop their microservice in. This article has an in-depth explanation of how Macaw supports this and what it takes to develop such microservice.

The impl/src/main/resources directory has a service-info.xml. This file contains the details about the service including the namespace, name and version of the service. Each service can have service specific configurations that can be listed in this file, so that those configurations are available to the service at runtime. Please read the service-info.xmlsection for more details.

Up till now the documentation has covered extensively how to get a basic microservice up and running. Let’s take a look at some service development recipes usable for developing non-trivial services.

MDR (Meta Data Repository) and Docker Registry together play a key role in the Macaw platform. MDR holds the Service Blueprints, Meta Data information, available Docker tags for a specific service. Docker Registry holds the service container images. In Macaw platform these two components go together and provide the end to end functionality of Service Provisioning.

MDR and Docker Registry information is provided as a configuration to Macaw Service Provisioner. Macaw Console interacts with the service provisioner and provides the ability to query a specific MDR/Docker Registry pair for available Blueprints and Services. Below is how a MDR/Docker pair is configured in the Service Provisioner.

"repositories": [
 {
 "name": "Macaw",
 "description": "Macaw MDR and Docker Registry Repository",
 "docker-registry": {
 "name": "Macaw Docker Registry",
 "description": "This is the Macaw docker registry",
 "host": "registry.macaw.io",
 "port": 443,
 "protocol": "https",
 "username": "macawio",
 "password": "rTVOJIqBYihZe4Xz",
 "email": "macaw@www.macaw.io"
 },
 "mdr": {
 "name": "Macaw MDR",
 "description": "This is the Macaw MDR",
 "host": "registry.macaw.io",
 "port": 8639,
 "protocol": "https",
 "token": "cd4111c5-e6eb-4fc4-b848-1ac9f72eda31",
 "version": "v2",
 "repo": "production"
 }
 },
 {
 "name": "onprem MDR/Docker",
 "description": "onprem MDR/Docker",
 "mdr": {
 "protocol": "http",
 "name": "macaw local MDR",
 "token": "9ba77ea5-2186-5282-9b88-93b373a59f31",
 "repo": "dev",
 "host": "platform-190.qa.macaw.io",
 "version": "v2",
 "port": 8637,
 "description": "This is the Locally deployed MDR"
 },
 "docker-registry": {
 "username": "macaw",
 "protocol": "https",
 "name": "macaw local Docker Registry",
 "port": 5000,
 "host": "platform-190.qa.macaw.io",
 "password": "macaw@local",
 "email": "macaw@local.com",
 "description": "This is the Locally deployed docker registry"
 }
 }
 ]

Note: Multiple MDR/Docker Registry pairs can be configured as shown above. The first MDR/Docker Registry should always be 
pointing to Macaw MDR/Registry.  This is auto-configured as part of the macaw platform installation. Also the Macaw platform 
provides the capability to install an on-prem MDR/Docker Registry and auto-updates the provisioner configuration.

MDR stores the Service Meta Data as deployable blueprints’ information. Macaw console queries the meta data repository server and fetches the available service blueprints that can be deployed. When Macaw platform is installed, the macaw-console is pre-provisioned with the Macaw central MDR information. This provides the ability to query Macaw MDR for any deployable blueprints and be able to deploy services locally on service hosts. For those who are developing microservices using the macaw platform it would be necessary to create custom blueprints and publish them to the MDR. The Macaw MDR is a read-only repository and does not allow publishing. The platform supports the ability to have multiple MDRs and gives the option to the end user to choose which MDR would like to be queried for the available blueprints. Platform provides MDR as one of the tools that can be deployed and configured locally. Once this is done as per the instructions, blueprints and service meta data can be published to the MDR via macawpublish tools (Refer to macawpublish tools documentation).

Below details help in installing a local MDR and configure it.

macaw tools install --tag <> --service macaw-mdr

Once you issue the above command, it would pull the supported MDR for the specific platform version you are running. Tag depends on the platform version you are running. Below is a typical MDR provisioning log.

The output of the MDR installation also provides additional configuration details needed for macawpublish tools. The file /opt/macaw-config/macaw-tools/macawpublish.globals provides the configuration required to be able to publish blueprints/service meta data to this MDR. This easy installation of MDR doesn’t provide the SSL endpoint for the MDR. Macaw platform supports SSL and non-SSL between macaw console and MDR.

Note: This specific MDR configuration configuration file would be referred to in the macawpublish 
section of the documentation.

For more details on the Docker Registry, refer to the documentation from Docker.

Macaw platform provides the ability to install a local docker registry with SSL enabled. Be aware that this may not be in line with the production configuration recommended for Docker Registry.

Below is the command which would help in installing a local docker registry using the macaw tool.

macaw tools install --tag 2.3.1 --service docker-registry

Note: Docker Registry is pulled directly from the Docker. The qualified registry with macaw platform is version 2.3.1. It is highly recommended to use the same version as the tag.

Post-installation the detailed instructions are provided on how to configure/enable service hosts to talk to on-prem Docker Registry. This additional configuration is needed as the Docker Registry is SSL enabled and the above installation has used self-signed certificates. Being self-signed, the certificates are not trusted by the docker and explicitly need to be added to the trust store.

Once both MDR and Docker Registry are installed, the MDR/Docker Pair configuration is automatically added to the Macaw Service Provisioner and would be visible under the repos in the macaw console. View macaw console documentation on how to view/provision Service Blueprints from different MDR/Docker Registry repositories.

Macawpublish is part of the Macaw SDK which provides the Service meta data/Docker Image/Blueprint publishing functionality to the developer. This part of the documentation provides all the details and capabilities of the macaw publish tool.

Directory Structure of the Tool

macawpublish comes default with the Macaw SDK and is located in the directory macaw-sdk/tools/macaw-publish-tools. Below is the directory structure of the Macaw publish tools.

--bin

This directory contains the python based macawpublish script.

--certs

This directory is a placeholder to hold certificates for MDR/Docker Registry.

--schema

This directory contains the Schema definition for a service blue print. When a blueprint is being 
published, it is verified against this schema.

--service-blueprints

This is a placeholder for service blueprints. If macawpublish tool is used to auto generate a blueprint, 
the generated blueprints are placed in this directory.

--service-lists

This contains files having list of services. The macawpublish tool can accept a file with list of files 
and be able to publish service meta data/docker images one by one in the order specified in the file.

--macawpublish.globals

This is the configuration file which provides the MDR/Docker Registry end points information to the 
macawpublish script. It is this file that need to be modified to tune to onprem MDR/Docker Registry.

MACAW_SERVICES_HOME & MACAW_SDK_HOME Environment Variables

macawpublish tries to locate service directories with reference to an environment variable “MACAW_SERVICES_HOME”. Lets assume the structure below is what the user has for microservices development.

/Users/foobar/Projects/macaw/microservices

— service1-dir

— service2-dir

— service3-dir

— sdk (This is SDK tar bundle is untar’ed)

For the above structure, MACAW_SERVICES_HOME environment variable can be set to /Users/foobar/Projects/macaw/microservices. From there onwards the user can publish services by simply using its directory name service1-dir.

MACAW_SDK_HOME refers to the directory in which the macaw SDK is residing.
MACAW_SERVICES_HOME refers to the directory where you are developing your microservices 

export MACAW_SDK_HOME=/Users/foobar/Projects/macaw/sdk/macaw-sdk
export MACAW_SERVICES_HOME=/Users/foobar/Projects/macaw/microservices
export PATH=$MACAW_SDK_HOME/tools/macaw-publish-tools/bin:$PATH

Note: Copy the above settings to your .bashrc so that they will always be sourced.

From here onwards the following documentation assumes that the user has set the environment properly.

Macawpublish relies on a single configuration file macawpublish.globals. This file provides the MDR/Docker Registry endpoints and the associated configuration for each. This part of the documentation helps in understanding the various configuration items. Some of the documentation is also part of the same file in the SDK as comments.

macawpublish.globals Location

The tool looks for this file under multiple locations with below preference.

• If an environment variable, MACAW_MDR_GLOBALS_FILE is set and pointing to a file, it is used.
• Else, it looks for “macawpublish.globals” in the user HOME directory which is typically C:\Users\<userid> or /home/<user id> or /Users/<user id> on Linux/OSX.
• Else, it uses the default file which is shipped with the macaw SDK. Note that the default file doesnt provide any default configuration, other than the documentation.

macawpublish.globals Configuration

[DEFAULT]
codebase=%(MACAW_SERVICES_HOME)s

[mdr]
#MDR Specific global properties. No need to change.
[docker]
#Docker Specific global properties. No need to change.

#Please refer to MDR documentation for more details on how MDR is structured. 
#With in MDR, there is a concept of repo. Repo can be any of the following supported values
#production, staging, qa, dev, demo.
#You need to define MDR/Docker properties for a given repo. There is possibility that same MDR can
#be holding multiple repos.

[dev]

#MDR Endpoint details.
mdr.endpoint=https://<FQDN/IP of MDR>:8639/mdr/service/macaw/execModule/mdr/
mdr.token=<RW Token for MDR>
#Repo in MDR.
mdr.repo.name=dev
#These are default settings. Need not be touched. For every that gets published to mdr/docker
#a TAG is created. Labels are assigned to tags for easy filtering. These are supported labels for 
#the tag definitions.
mdr.tag.label.allowed=INTERNAL,EXTERNAL,DEMO,PRODUCTION,QA,DEV,STAGING
mdr.tag.label.default=INTERNAL

#Below is optional. If you want to enforce strict SSL, then point to the root ca cert.
#This is for verifying the certiticate checking. This is PEM format of CA key and cert.
# cat ca.key ca.crt | tree ca.pem
#Either fully qualified location can be specified or provide path with reference to 
#macaw-sdk/tools/macaw-publish-tools/certs
#In the below the certificate is looked in the above directory.
#mdr.certificate=ca.pem

#Docker Registry Endpoint details.
docker.registry.host=<Docker End Host>
docker.registry.port=<Docker Port>
docker.registry.username=<Username>
docker.registry.password=<Password>
docker.registry.email=<Email. Email is deprecated for latest docker release. You can provide a dummy email to maintain backwards compatibility>

#The below settings can be configured to enable remote publishing of docker images.
#On some developer machines, there is a possibility of docker not being present. 
#In this case, the developer can chose a remote linux with docker publish capabilities.
build.server=
build.server.user=
#Key to be used for password-less SSH.
build.server.user.key=
#Provide a specific key file for password less SSH. If any of these are not specified, then default SSH keys are attempted.
build.server.user.key.file=
#To use a fixed password. This is plain text password and discouraged.
build.server.user.password=

Note: When MDR/Docker tools are installed using Macaw tools install <>, the configuration for MDR is auto-generated. The configuration can be copied to $HOME directory on the development machine to be able to interact with the deployed MDR while publishing services.

# cd $MACAW_SDK_HOME/tools/macaw-publish-tools/
# sftp macaw@<Platform IP/FQDN>:/opt/macaw-config/macaw-tools/*.globals .
password: *********

Note: By default, for MDR/Docker Tools, the configuration is placed under /opt/macaw-config/macaw-tools/ on the platform VM.

Certificates for Installed Docker Registry

Macaw platform tools by default use self-signed certificates. The self-signed certificates can be replaced with trusted certificates by following the documentation on how to replace SSL certificates in Macaw platform.

Since the certificates are self-signed, docker hosts will not trust the on-prem docker registry by default. For this reason, we need to add the trusted certificate of the Macaw platform to the docker trusted store on the host from where services will be built/published.

Below are the 2 methods that are known to work as per standard.

Download the ca.crt from the machine where Macaw platform was installed. The default location of the ca.crt is /opt/macaw-config/certificates/ca/ca.crt

Method 1 – Works for CentOS/Redhat Linux Hosts

Docker Registry is installed with self signed private certificate. 
Below steps/configuration are needed for the any docker host to be able to talk to this registry.

Step 1: Login to the docker Host(s) and execute the below 

        sudo mkdir -p /etc/docker/certs.d/<Platform IP/FQDN>:5000

Step 2: Download ca.crt from the platform host to the docker host(s). 

        sudo scp macaw@<Platform IP/FQDN>:/opt/macaw-config/certificates/ca/ca.crt /etc/docker/certs.d/<Platform IP/FQDN>:5000

If you are using the platform host as Remote Docker Build server, you can simply do the below, on the platform Host.

sudo cp /opt/macaw-config/certificates/ca/ca.crt /etc/docker/certs.d/<Platform IP/FQDN>:5000

Step 3: Verify docker login

        docker login -u macaw -p macaw@local -e macaw@local.com <Platform IP/FQDN>:5000

Method 2 – For Mac

Step 1: Download the ca.crt in to the ~/Documents folder.

sftp macaw@<Platform VM IP or FQDN>:/opt/macaw-config/certificates/ca/ca.crt ~/Documents/
Step 2: Connect to the Docker VM using the below command

screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

Step 3:

mkdir -p /etc/docker/certs.d/<Platform VM IP or FQDN>:5000  cp /Users/<id>/Documents/ca.crt /etc/docker/certs.d/<Platform VM IP or FQDN>:5000/ca.crt

Step 4: Verify docker login  

docker login -u macaw -p macaw@local -e macaw@local.com <Platform VM IP or FQDN>:5000

Note: When the Docker is restarted or MAC is restarted, this change is gone and needs to be repeated again. This is a Docker limitation on Mac currently.

Once the above steps are complete, the machine will now be able to talk to the on-prem installed docker registry w/o SSL issues.

Macawpublish tool requires the presence of Docker on the local development machine to be able to push service container images to the Docker Registry. This would also require configuration of Docker Daemon on local host w.r.t certificates.

This restriction can be avoided on the local development machine by enabling remote publishing in macawpublish tool. With remote publishing none of the instructions and functionality of the macawpublish tool would change, except that now the service artifacts necessary for the containerization of the service are uploaded to a remote build server and the docker build/tag/publish operations are executed on the remote server. This avoids having Docker install on your local machine.

The remote build server should be configured with proper certificates and access to the on-prem registry installed as part of the macaw tools. Refer to MDR/Docker Section above for details on how to achieve this.

Note: For quick development purpose, you can use your platform instance as a remote build server.

Enabling Remote Publishing

Edit your macawpublish.globals property file and add the below entries.

[dev]
...
...
..
build.server = <IP/FQDN of Remote Server>
build.server.user = <User>
build.server.user.password=<password>

Providing a password in plain text is not considered safe. The user can enable Password-Less SSH by executing the below commands. The macawpublish tool automatically generates a build key and sets password-less SSH for this key.

python macawpublish --repo dev deploy-ssh-keys --user cfx --host 10.95.101.231

The above command automates the process and updates the macawpublish.globals with the key information.

If password-less SSH is already setup between the local machine and the remote build server with user’s default SSH Keys, specify the below properties.

[dev]
...
...
..
build.server = <IP/FQDN of Remote Server>
build.server.user = <User>

In cloud instances like Amazon/Oracle/Azure, password-based authentication may not be allowed. In this case the user may already have a private key to login to the instance. Follow the below configuration for this case.

[dev]
...
...
..
build.server = <IP/FQDN of Remote Server>
build.server.user = <User>
build.server.user.key.file=<Fully Qualified Location of the private key being used to login to the instance as the above user>

If the platform instance where macaw tools are installed is in a cloud like AWS, Azure, refer to the below details on how MDR/Docker Pair interactions work w.r.t public/private address of the platform instance.

When the platform instance is in a public cloud, typically the user would be having a private IP/DNS. and Public IP/DNS. For all the platform configuration via macaw setup etc., the user would be using the private IP/DNS. When it comes to interacting with the platform instance from the external world the user would be interacting via the public IP/DNS.  To enable this mixed IP interactions, follow below guidelines in enabling the platform instance as remote docker build server.

After installing the MDR tools, macawpublish.globals is automatically created and this can be used directly as long as the user is communicating only via private IP addresses. If intending to interact with the instance from a local laptop/PC, the user must interact via public IP. In this case, change the macawpublish.globals like below to enable remote docker publishing.

Auto Generated macawpublish.globals

[dev]

mdr.endpoint=http://10.0.0.4:8637/mdr/service/macaw/execModule/mdr/
mdr.token=5f35c8df-813b-5d11-a156-53a5e41b1332
mdr.tag.label.allowed=INTERNAL,EXTERNAL,DEMO,PRODUCTION,QA,DEV,STAGING
mdr.tag.label.default=INTERNAL
mdr.repo.name = dev
#mdr.certificate=<fully qualified location. Ex: %(codebase)s/macaw-skd/macaw-publish-tools/certificates/ca.pem>
#Docker Registry Endpoint details.
docker.registry.host=10.0.0.4
docker.registry.port=5000
docker.registry.username=macaw
docker.registry.password=macaw@local
docker.registry.email=macaw@local.com

[dev]

mdr.endpoint=http://104.211.243.97:8637/mdr/service/macaw/execModule/mdr/
mdr.token=5f35c8df-813b-5d11-a156-53a5e41b1332
mdr.tag.label.allowed=INTERNAL,EXTERNAL,DEMO,PRODUCTION,QA,DEV,STAGING
mdr.tag.label.default=INTERNAL
mdr.repo.name = dev
#mdr.certificate=<fully qualified location. Ex: %(codebase)s/macaw-skd/macaw-publish-tools/certificates/ca.pem>
#Docker Registry Endpoint details.
docker.registry.host=10.0.0.4
docker.registry.port=5000
docker.registry.username=macaw
docker.registry.password=macaw@local
docker.registry.email=macaw@local.com

The user needs to replace the MDR endpoint with the public IP of the instance. Since the user is going to use the platform instance as remote docker build server,
the docker host does not need to be changed.

Once the changes described above to macawpublish.globals are executed, simply enable the remote build using the instructions specified for enabling remote docker build server.

This part of the documentations provides the Docker Image and Blueprint publishing details for a Macaw enabled Microservice. The below steps are typically performed on the development machine where microservices development takes place.

Before publishing an image and metadata for service, publishing the blueprint for the service is important. Without the blueprint, Macaw console will not have the knowledge about the service Docker image or its metadata.

Blueprint can be written manually by following the schema or can be created automatically. This part of the document explains the automated creation of the blueprint using macawpublish tool. Refer to full documentation of Service Blueprints to fine-tune and add more capabilities to the blueprint.

macawpublish create --help

usage: macawpublish create [-h] [--list SERVICE_DIRS_FILE] [--debug]
                           [service-dirs [service-dirs ...]]
positional arguments:
  service-dirs          Service directories - Either name or full path. If
                        only name is provided, then $env(MACAW_SERVICES_HOME) is used to
                        prepend
optional arguments:
  -h, --help            show this help message and exit
  --list SERVICE_DIRS_FILE
                        File which is containing service directories (name or
                        full path). One service directory per line. This
                        enables publishing multiple services in one command.
  --debug               Enable debugs

Creating a blueprint with a single service

macawpublish create service1-dir

Example:

macawpublish create calculator

Creating a blueprint with multiple services

A blueprint is not limited to a single service. It can include multiple services. When multiple services are part of the blueprint, by provisioning the blueprint, the user will be provisioning all the services specified in the blueprint.

macawpublish create service1-dir service2-dir

Example:

macawpublish create calculator todo-list

To publish the blue print to MDR, execute the below command.

macawpublish blueprint <Location of the Blueprint JSON>

After creating and publishing blueprints, publishing your container images and meta data for Microservices may begin. Below are the simple steps to publish an image and meta data to the MDR/Docker Registry.

Before publishing the service to MDR/Docker registry, make sure to compile the <icroservice so that the necessary artifacts are generated which need to be bundled into the container image. Refer to the SDK documentation on how to compile the Microservice.

macawpublish service <service-dir1> --tag <tag>
or
macawpublish service <service-dir1> <service-dir2> --tag <tag>
or 
macawpublish service --list <File containing list of Service Directory names or fully qualified director paths> --tag <tag>

Once the blueprint and image/meta data is published to MDR/Docker pair, the user may provision service(s) through the Macaw Console. Refer to the Macaw Console documentation on how to locate the blueprint and provision services.

This part of the documentation provides Macaw support for auto-deploying of war files.

In Macaw platform, web applicationss (war files) can be packaged as standard Docker containers and deployed into existing Tomcat instances running as Macaw UI Pairs. Refer to macaw UI pairs documentation on how to create and provision UI Pairs.

macawpublish webapp --file <War File Location> --name <Name of the WebApp> --version <Version of the WebAppn> --tag <Docker/MDR Tag>

Macaw also supports exporting the custom icon from existing web application to the launchpad so that is visible on the Web Server home page and users can click and access the web app. By default Macaw provides a default icon.

To provide a custom icon, use the below command.

macawpublish webapp --file <War File Location> --name <Name of the WebApp> --version <Version of the WebAppn> --tag <Docker/MDR Tag> --icon <Icon Path relative to your web app>

Note: The Icon path should be relative to your web app path. Lets say if you icons are packaged in your app under icons/logos/mylogo.png, you provide "--icon icons/logos/mylogo.png"

Once the web application container is published, to deploy the web application into an existing Macaw UI pair, create a Blueprint and publish the blueprint to the MDR.  These steps show how to create a simple web application blueprint.

./macawpublish create-webapp --name <Name of the WebApp> --version <Version of the WebAppn>

Note: Make sure the name and version are the same as what is used for publishing the web application. The name and version together form the Docker container repository.

Once the blueprint is created, publish the blueprint to the MDR.

./macawpublish blueprint <Location of the Blueprint>

Below is an example of 2 steps being done.

$ ./macawpublish create-webapp --name sample --version 1.0.0
2017-01-24 12:38:29,020 [macawpublish] INFO     - MACAW_SERVICES_HOME is pointing to: /Users/ravjanga/Documents/CFXDev/SDK/macaw-sdk-0.9.1-SNAPSHOT/quickstarts
2017-01-24 12:38:29,020 [macawpublish] INFO     - Generating blueprint for sample-1.0.0

Blueprint create successfully and stored at: /Users/ravjanga/Documents/CFXDev/macaw-sdk/tools/macaw-publish-tools/bin/../service-blueprints/macaw-webapp-bp-e233fe20-cab1-5b59-b700-2fc6ebde3754.json
Blueprint can be uploaded to MDR using macawpublish blueprint 

$ ./macawpublish blueprint macaw-webapp-bp-e233fe20-cab1-5b59-b700-2fc6ebde3754.json
2017-01-24 12:38:37,418 [macawpublish] INFO     - MACAW_SERVICES_HOME is pointing to: /Users/ravjanga/Documents/CFXDev/SDK/macaw-sdk-0.9.1-SNAPSHOT/quickstarts
2017-01-24 12:38:38,031 [macawpublish] INFO     - Published to MDR successfully. Tag: latest, Artifact: e233fe20-cab1-5b59-b700-2fc6ebde3754, Repo: dev
2017-01-24 12:38:38,032 [macawpublish] INFO     - Blueprint macaw-webapp-bp-e233fe20-cab1-5b59-b700-2fc6ebde3754.json published successfully.

Dashboard is the default landing page, after user login into Macaw Console portal.

The following details are presented in Main Dashboard

• Counts – Total Count of all:
  • Service Groups
  • Platform Essential Service Clusters
  • Other Service Clusters
  • Web Applications
• Doughnut graphs depicting the status of all Instances under:
  • Platform Service Instances
  • Other Service Instances
  • Web Applications
• List of all Service Groups with Environment name/type, total number of services/webapps present in each group. On selecting the group, ServiceGroup Dashboard is displayed. This dashboard offers the same options, as those shown under ServiceGroupDashboard of ServiceManager -> ServiceGroups & Applications.
• List of all Service Clusters with name, namespace, version, environment name/type, total/available instances of the cluster. On selecting any cluster, ClusterDashboard is displayed. This dashboard offers same options, as those shown under ServiceClusterDashboard of ServiceManager -> ServiceGroups & Applications.

The Service Manager Option of the DevOps Console provides a comprehensive functionality pertaining to Macaw Microservices. Each section displays one unique functionality that can be performed using this UI.

Service Groups & Application
This page lists all the ServiceGroups present and provides various options on the ServiceGroup, its Service Clusters, and Web Applications

Service Group Options:

• ‘View Details’ presents the Service Group Dashboard
• ‘Delete’ option can be used to deprovision all the instances of all service clusters and web applications in the group.

Service Cluster Options:

• ‘View Details’ of the Service Cluster presents the Cluster dashboard
• ‘BrowseAPI’ navigates to the APIBrowser of selected service cluster
• ‘Expand Cluster’ can be used to add new instances to the cluster, with Environment-type ‘Standalone-docker’.
• ‘Projects’ option navigates to the projects page, which specifies the list of projects, that can access this ServiceCluster. ‘Add’ link can be used to add more projects to this list.
• ‘Show Metrics’ displays the metrics of Memory and RPC Requests made.
• ‘Rolling Update’ can be used for a kubernetes cluster, to upgrade the current instance(s)
• ‘Scale Up’ of a kubernetes cluster, adds new instance to the cluster
• ‘Scale Down’ of a kubernetes cluster, deprovisions an instance from the cluster

Web Application Options:

•  View Details option gives the dashboard of the application

Service Group Dashboard:

Service Group Dashboard provides the following details

• Total number of Service Clusters and Web Applications present in the group
• Doughnut chart of Service Instances and Web Applications.Note:-Doughnut charts depict the current states (‘Available’, ‘Paused’, ‘UnReachable’, ‘UnAvailable’) of all instances present in the group
• The list of all ServiceClusters/WebApplications, along with other details like Namespace, Version, Environment, EnvironmentType, Total instances in the cluster, Total available instances in the cluster

Delete option in the menu of the Group can be used, to delete all the instances of all listed clusters. Selecting any service cluster, presents the Cluster Dashboard.

Service Cluster Dashboard:

Cluster Dashboard provides the following details

• Total number of instances present in the cluster, total available instances in the cluster.
• Service Instance information like Container Id, Instance Id, State of the Instance, Image Tag used for provisioning the instance

At cluster level, the options provided are BrowseAPIs, ExpandCluster, Projects, ShowMetrics, Rolling Update, Scale Up, Scale Down.

Instance level options:

• View Details gives more details of the Instance like Name, Namespace, Version, Host where the instance is deployed, Registered time, Repository selected during deployment etc
  
• Pause/Resume an Instance
• Deprovision a Standalone-docker cluster instance
• ‘View Docker Info'(for Standalone-docker) / ‘View Info'(for Kubernetes) of the instance
• View ESLogs
• Show Metrics of the instance
• Launch ADPM with selected instance as default value for agent. This option is enabled if it is configured during deployment.

ShowMetrics at Cluster level

Duration and Instance can be changed to get the metrics for required time period and instance.
Instance level ‘show metrics’, gives metrics of the selected instance

Docker-info of a standalone-docker instance

‘Details’ tab gives the ‘docker inspect’ of the instance container. ‘Logs’ tab gives the ‘docker logs’.

Expand Cluster form of docker standalone clusters

Options are given to select Deployment Flavor & Image tag while Expanding cluster.

Scale Up form of Kubernetes cluster

Number of instances to be added to the Kubernetes cluster, can be selected here. Cluster is allowed to have a total of 8 instances

ScaleDown form of Kubernetes Cluster

Number of instances to be scaled down can be selected.

Service Clusters
This page provides the overall cluster view of all Service Clusters

The menu of any ServiceCluster is same, as the cluster level menu seen in the ServiceGroups & Applications page

Service Catalog
Service Catalog provides the blueprints that wrap various microservices. These microservices are internally mapped to docker containers and are published via blueprints. Blueprints are categorized based on the underlying functionality of services or set-of-services. These blueprints are presented in two views (a) Card layout (b) List layout.

The repository and the blueprint tag have to be selected.

Using the catalog card layout section or list layout section, services can be deployed onto per-configured environments. To deploy service(s) from any blueprint, the below required information need to be provided.

• Service Information
  • Select the project which requires access to the service(s)
  • Select ‘Delete Service Metadata’, if any metadata of service(s)(which were previously deployed) has to be cleared
• Policies
  • Select ‘HA Deployment Policy’
  • Select Deployment flavor. Each flavor is tied to its respective resource-profile(defined in the blueprint json file)
  • Select the required ‘Environment Add on Features’ that have to be applied to the service(s)
    • Macaw Debug: to enable debugging on the service
    • APM: When this option is enabled, the instance(s) of the service(s) are tracked for performance management. They Performace Manager can be opened from <host-url>/adpm or using Launch ADPM option of the service instance.
    • Host Logging: When enabled, the docker logs of the service(s) are written to be host
    • Eslogs:When enabled, the ESLogs are provided under the ‘Vew ESLogs’ of the instance(s).
    • Native Memory Tracking Detail/Summary: To enable Native Memory Tracking flag on JVM
• Instance Counts
  • Select the number of instance(s) required for each Service Cluster
  • Select the image tag to be used to deploy the service(s)
  • Reset to Default/Reset to Minimum Quantities can be used to reset the number of instances
  • For Web Application, ui-pair has to be selected
  • ‘Verify’ option can be used to check if the fields are properly populated

Deployment Status
Deployment Status provides the following details of all the deployments done

• Blueprint used in the request
• ‘Project’ for provisioning/deprovisioning requests shows, the project which is selected during deployment of service(s)
• Request Type can be ‘Provisioning Services’, ‘Deprovisioning Services’, ‘Expand Service Instance Cluster’, ‘Scale Up Service Instance Cluster’, ‘Scale Down Service Instance Cluster’ depending on the deployment request
• ‘Successful’, ‘In Progress’ or ‘Failed’ status is given under Request Status
• Start Time/End Times of the request

‘View Details’ presents more details about each request, depending on the request type

Sample Provisioning request details

Sample Deprovisioning request details

API Browser
API Browser provides the functionality to Post RPCs of various Service Clusters, in different tabs

New services can be added to the tab, by using ‘Add new service tab’ option on the right side
Each service tab in browser, provides the below functionality

•  Displays name, namespace, version, clusterId of the Service Cluster
• RPC selector to select the required RPC from the dropdown
•  Once an RPC is selected, ‘RPC Info’ option is enabled. It provides
  • description of the RPC
  • input parameters along with their data types
  • end point url
• ‘Format Input’ icon can be used to format the json input
• Json validation is done for the given input. If input given is not in proper json format, cross mark appears on the line of wrong input
• RPC output is displayed along with its Duration. It shows the time taken by API Browser to post RPC and give the output, along with the actual API Response time
• ‘Copy to Clipboard’ option is provided for the output field
• ‘Service History’ option on the right side, provides details of the RPCs executed.
  • Unique combinations of RPCs executed and the input given to them, are entered in the History
  • The time of last RPC call with a particular input, is displayed. If an RPC is executed more than once, with same input, then the time is updated for that respective entry
  • Red/Green colors in the history indicate, whether the RPC request was successful or failure
  • Each entry in the history can be removed separately or ‘Clear History’ can be used to remove all the entries from history of the current Service Cluster
  • Search field can be used to search entries, with RPC names or RPC input given
• Service Documentation can be referred for the following details
  • List of RPCs with description and input parameters
  • List of Notifications that get published by the Service, along with their description and inputs
  • List of Objects used by the service, their description and json structure

RPC Info option

RPC History

Service Documentation

Environments
This page displays Provisioning Environments that were configured in the platform

Each Standalone-docker environment displays the host(s), that are used for provisioning macaw microservices. Kubernetes environment gives the Kubernetes master URL. ‘View Details’ presents the ‘Environment Dashboard’ for the selected environment.

Environment Dashboard

It contains the following information

• Total Service Clusters/Web Applications deployed in the selected environment
• Doughnut Chart(s) of Service Instances/Web Applications(if any) depicting their states
• List of all Service Clusters and WebApplications along with other information like Name, Namespace, Version, Environment, Total/Available Instances. On Selecting any cluster, the respective Cluster Dashboard is displayed, along with the actions that can be peformed in ClusterDashboards.
• List of all Environment Repositories, that are configured in the platform for current environment.

Service Registry Metrics
Metrics graphs on Service Registry are provided, for the selected duration on

• Memory – jvmMaxMemory, jvmTotalMemory, jvmFreeMemory

• Requests – Successful Service Registration Requests Counter, Total Service Registration Requests Counter

• Total ServiceAPI Queries Received Counter

Administration UI provides various administrative functionalities to Platform Administrators as shown in the following picture:

Tenants
Details of all tenants are displayed in the ‘Tenants’ section. The following actions can be performed on tenants:

• Add a new tenant using ‘Add’ link
• Activate/Deactivate the tenant
• View Summary of the tenant
• Update the details of tenant
• Navigate to TenantAdministrators page
• Navigate to RootOrganization page
• Delete non-default tenants

Organizations
Every tenant has a Root Organization

Following actions can be performed on sub-organizations

• New sub-organizations can be added under the Root Organization of non-default tenants. Under each sub-organization, more sub-organizations can be added.
• Activate/Deactivate the sub-organization
• View Summary
• Update the sub-organization
• Delete sub-organization. A sub-organization can’t be deleted if it has sub-organizations under it.
• Navigate to Projects page of this organization.

Projects:

Each Organization has a default project. Following actions can be performed on Projects

• Add a new project from the ‘Add’ link
• Activate/Deactivate non-default projects
• View Summary of the project
• Update the Project
• Delete non-default projects

TenantAdministrators

Details of all the tenant administrators of a tenant, are displayed in this page.Following actions can be performed on tenant administrators

• Add a new tenant admin from the ‘Add’ link
• Activate/Deactivate the user
• View Summary
• Update the details of admin
• Reset the current password of admin
• Delete the tenant admin

Platform Administrators

Following actions can be performed on Platform Administrators

• Add a new PlatformAdmin
• Activate/Deactivate other users
• View Summary of the user
• Update the user
• Reset Password of the user
• Delete other users

Events are captured whenever any RPC of a service is executed. ServiceEvents presents the metrics of these events, along with the details of failed/slow Invocations.

Configuration:
The Configuration section contains the list of all the services with name, namespace, version. Under each service,

• There is the configuration to get the details of failed/slow Invocations of the service RPCs.
• View Events/Metrics of all the executed RPCs of service.

Configure Metrics:

  ‘AddConfig’ link can be used to add configuration for any RPC of the service

Add RPC configuration form inputs:

• Select RPCs from the list, for which the configuration has to be done.
• ‘High Response Time’ field is used to get the details of slow invocations. If the RPC takes more time to execute than the value give here, then it is considered as Slow Invocation and its details are captured.
• ‘Sample Errors’ field is used ,to enable capturing the details of failed invocations. If the RPC execution fails, then the details of this invocation are captured.
• ‘Maximum Sampled Requests per Minute’ represents, maximum number of failed/slow invocations of the RPC(per minute), for which details have to be captured. If there are more invocations that are slow/failed, than the number specified here, then the details of remaining invocations are discarded.

Added configurations can be deleted/modified.

View Events/Metrics:
‘VewEvents/Metrics’ presents all the invocations of any RPC of the service, along with the metrics of each RPC.

Summary of fields:

• ServiceVersion: The version of the service used
• MethodName: RPC of the current service,which is executed
• InvocationCount: Number of times, this RPC is called
• Throughput(rpm): Takes into account, the total time taken by all invocations and the total count of invocations, to give the estimation of , how many invocations can be completed in a minute.
• ErrorCount: Total number of failed Invocations
• ErrorRate %: Percentage of failed Invocations, among the total invocations of the RPC
• Min(ms): Minimum server time taken to complete the RPC request, among all the invocations
• Avg(ms): Average of the server times of all invocations
• Max(ms): Maximum server time taken among all the invocations

Event Metrics:
Invocation Metrics gives the list of RPCs executed of all the services, along with the metrics, invocation details of failed/slow Invocations.

Failed Invocations:
If an RPC execution fails, the details of the failed invocation are listed under ‘Failed Invocations’. These details are captured, only if the RPC is configured to get the details of failed invocations.

Summary of fields:

• Correlation Id: Id of the correlation, which has the current invocation of the RPC
• StartTime: Time at which the invocation started
• Client ID: Client instance ID which called this RPC
• Server ID: ID of the instance(in the service cluster) which executed this method
• Service API: Service and RPC names
• ServerTime(ms): Time taken by server to complete the current failed invocation
• RoundTripTime(ms):Complete time taken in the client-server interaction

Interactions of failed invocations

The above sequence diagram shows all the invocations which are involved in the current correlation:

InteractionDetails of the correlation, contain the following fields

• Type of event(REQUEST/RESPONSE/NOTIFICATION)
• Instance Ids, Names of client, server instances
• SendTime,ReceiveTimes of the event
• Outcome of Request event and notifications, is ‘invoked’. Outcome of Response event shows failure.
• Total time taken at server side, roundtrip time to complete the RPC request

View ESLogs of failed invocations
ESLogs related to the current failed invocation are shown here

Slow Invocations
Depending on the configuration made for an RPC for slow invocations, the details of successful but slow invocations are captured here.Fields of this report are similar to the fields of failed invocations.

Interactions & Details of slow invocations

ESLogs of slow invocations
ESLogs related to the current slow invocation are shown here.

View Summary

View Summary gives the metrics of RPC invocations. Median, 95 Percentile, 99 Percentile values are calculated using server times of all invocations of the RPC.

This part of the document provides detailed instructions on how to enable A local development environment on a laptop/PC and assumes that platform installation is already done and running.

A Developer using the Macaw SDK must have the following software installed in order to get started developing microservices.

• Oracle JDK, versions 8 and above
• JAVA_HOME env variable should be setup to point to this JDK installation.
• Apache Ant 1.9.x (and above) with Ant-contrib.
• Python 2.7.X
• Docker version 1.11.x and above (Optiona)

Note: Refer to each OS section for more details on how to setup/install each of the above requirement.

The Java-based microservices projects generated by the code-generator tool are Eclipse-based and can be imported into a Eclipse installation. There is no Eclipse version dependency in the project. It should work for all recent Eclipse versions; Eclipse Mars or a later version is recommended. However even if you are a IntelliJ or Netbeans user, the project can be imported into a favorite IDE with almost no extra effort.

Follow the steps indexed here to install the necessary software dependencies for enabling macaw SDK development on a Windows PC/Laptop. The steps also guide through the compiling/publishing/provisioning of an example service which comes as part of the SDK.

Installation and Setup – Necessary Software Packages

Python Installation – 2.7.x

Download Python from the below we site and follow through the regular installation steps. This would install Python 2.7.x on the local machine. The default path of the Python installation is C:\Python27.

Download Link: https://www.python.org/ftp/python/2.7.13/python-2.7.13.msi

Ant Installation – 1.9.X

Download the ant ZIP from the below location. Right click on the ZIP and extract to C:\.

This would create the folder C:\apache-ant-1.9.7

Download Link: http://archive.apache.org/dist/ant/binaries/apache-ant-1.9.7-bin.zip

JDK Installation – 1.8.0.121

Download the JDK from the Oracle download site. Accept the license agreement and download the software. For Windows 64 bit, it is necessary to download, Windows x64 version of the JDK.

Download Link: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

For regular installation, the installer would install JDK in the below directory.

C:\Program Files\Java\jdk1.8.0_121

Setting up the environment

• Open a Windows Power Shell Window with Admin Permissions. By default power shell opens in non admin mode. Right click on the Power Shell icon and click on “Run As Administrator”
• Execute the below 3 commands to set the environment. Once it  is done, please close the Current Power Shell Window and Open a new Window.

[Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\Python27\;C:\Python27\Scripts\;C:\apache-ant-1.9.7\bin;C:\Program Files\Java\jdk1.8.0_121\bin", "Machine")

[Environment]::SetEnvironmentVariable("ANT_HOME", "C:\apache-ant-1.9.7", "Machine")

[Environment]::SetEnvironmentVariable("JAVA_HOME", "C:\Program Files\Java\jdk1.8.0_121", "Machine")

Note: If any different versions of Python, JDK, ANT are installed, please change the paths accordingly.

• Verify the installation with below commands. Open a new Power Shell Window with Admin Permissions like before, to do the below verification. Path/Environment changes will not be applicable to the current running power shell window.

PS C:\WINDOWS\system32> ant -version
Apache Ant(TM) version 1.9.7 compiled on April 9 2016
PS C:\WINDOWS\system32>
PS C:\WINDOWS\system32> java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
PS C:\WINDOWS\system32>
PS C:\WINDOWS\system32> python --version
Python 2.7.13
PS C:\WINDOWS\system32>

• Install Python modules – Open Power Shell window with Admin Permissions and run the below command.

python -m pip install requests==2.11.1 paramiko==2.0.0 jsonschema==2.5.1 tabulate==0.7.7

Macaw Environment Setup

SDK Download and Setup

• SDK Download – Download the macaw SDK (ZIP Version). The email invitation/guidelines would have the download link. 
• Unzip the SDK into the project directory. For example: C:\Users\foobar\Documents\foobar\. The SDK would be unzipped and show a folder structure C:\Users\foobar\Documents\foobar\macaw-sdk-<version>
• Macaw SDK would be requiring 2 environment variables to be able locate and identify the SDK run time libraries and to locate the microservices . Execute the below commands in a power shell window opened with admin permissions. Adjust the paths accordingly to the directory paths.

[Environment]::SetEnvironmentVariable("MACAW_SDK_HOME", "$ENV:HOMEDRIVE$ENV:HOMEPATH\Documents\foobar\macaw-sdk-<version>", "User")
[Environment]::SetEnvironmentVariable("MACAW_SERVICES_HOME", "$ENV:HOMEDRIVE$ENV:HOMEPATH\Documents\foobar", "User")

From here onwards, $ENV:MACAW_SERVICE_HOME would be the location under which macawtool would try to locate the services based on the name of the service folder. Note that this is not a strict limitation. If the structure below is followed, service artifacts can be easily published by the macaw tool, by just specifying the service directory name. If this procedure is not followed, then absolute path to the service must be specified. For details, read the Macaw Tool documentation provided at the end.

$ENV:MACAW_SERVICE_HOME/service1/api

$ENV:MACAW_SERVICE_HOME/service1/impl

$ENV:MACAW_SERVICE_HOME/service2/api

$ENV:MACAW_SERVICE_HOME/service2/impl

Note: Instead of HOMEDRIVE, HOMEPATH variables, you can provide the full path to your SDK and microservices directory.

Macaw Tool Setup – For Publishing

The below steps can only be performed, with a working platform instance or access to an existing platform installation.

• macaw publish tool is already bundled with the SDK. It is located at $ENV:MACAW_SDK_HOME\tools\macaw-publish-tools\bin. It is python script and can be executed using the python interpreter.
• macaw publish tool relies on a configuration file called “macawpublish.globals“.  The tool looks for this file under multiple locations with below preference.
  • If an environment variable, MACAW_MDR_GLOBALS_FILE is set and pointing to a file, it is used.
  • Else, it looks for “macawpublish.globals” in the user home directory which is typically C:\Users\<userid>.
  • Else, it uses the default file which is shipped with the macaw SDK. Note that the default file doesnt provide any default configuration, other than the documentation.
• Assuming that the complete setup of MDR/Docker Pair as part of the installation, follow the below instructions.
• Download/Create the “macawpublish.globals” from the platform VM Instance to the HOME directory C:\Users\<userid>.
  • On the platform VM, find the “macawpublish.globals” at the location

    /opt/macaw-config/macaw-tools/macawpublish.globals

• Once copied the “macawpublish.globals”, verify the connectivity to MDR/Docker Pair, using the below command.

cd $ENV:MACAW_SDK_HOME\tools\macaw-publish-tools\bin

python macawpublish verify

With this the PC/Laptop setup is finished for development environment. Refer to the below sections on how develop, publish and deploy microservices . To test the end to end environment, follow the next section which lists steps on compiling SDK bundled example services, publishing and deploying them.

MicroServices Development

Macaw Tool Documentation for Publishing Blueprints and Docker Images

Testing the End to End Environment Using Example Services in the SDK

Compiling

• Open a Power Shell Window with Admin permissions.
• Go to the below directory which has the example services bundled. Calculator service is used for this documentation purpose. Repeat the similar steps for other example services as well.

cd $ENV:MACAW_SDK_HOME\quickstarts
cd calculator\api
ant clean deploy
cd ..\impl\
ant clean deploy

Generating Blueprint and Publishing

Now generate a blueprint for calculator service and publish to MDR. The blueprint publishing doesn’t have to repeated unless there is a change in the blueprint. Likely it would be a 1 time publish, but for any changes, re-publish is necessary.

PS C:\macaw-sdk\quickstarts\calculator\impl> cd $ENV:MACAW_SDK_HOME\tools\macaw-publish-tools\bin
PS C:\macaw-sdk\tools\macaw-publish-tools\bin>
PS C:\macaw-sdk\tools\macaw-publish-tools\bin> python macawpublish create $ENV:MACAW_SDK_HOME\quickstarts\calculator

2017-01-24 15:36:40,957 [macawpublish] INFO - MACAW_SERVICES_HOME is pointing to: C:\macaw-sdk\quickstarts
2017-01-24 15:36:40,957 [macawpublish] INFO - Service Definitions: ['C:\\macaw-sdk\\quickstarts\\calculator']
2017-01-24 15:36:40,957 [macawpublish] INFO - START: processing service definition - calculator

Blueprint create successfully and stored at: C:\macaw-sdk\tools\macaw-publish-tools\bin\..\service-blueprints\macaw-service-bp-da777d9e-9d8d-5ef5-a9c7-e58d3ebe875f.json

Blueprint can be uploaded to MDR using macawpublish blueprint <Blueprint File>


PS C:\macaw-sdk\tools\macaw-publish-tools\bin> python macawpublish blueprint macaw-service-bp-da777d9e-9d8d-5ef5-a9c7-e58d3ebe875f.json

2017-01-24 15:37:05,144 [macawpublish] INFO - MACAW_SERVICES_HOME is pointing to: C:\macaw-sdk\quickstarts
2017-01-24 15:37:05,878 [macawpublish] INFO - Published to MDR successfully. Tag: latest, Artifact: da777d9e-9d8d-5ef5-a9c7-e58d3ebe875f, Repo: dev
2017-01-24 15:37:05,878 [macawpublish] INFO - Blueprint macaw-service-bp-da777d9e-9d8d-5ef5-a9c7-e58d3ebe875f.json published successfully.

PS C:\macaw-sdk\tools\macaw-publish-tools\bin>

Publishing Docker Image and Meta Data

Now publish the docker image and meta data for the calculator service. This is a step that will be repeated multiple times in service development.

PS C:\macaw-sdk\tools\macaw-publish-tools\bin> python .\macawpublish service --tag calc-demo $ENV:MACAW_SDK_HOME\quickstarts\calculator

2017-01-24 15:48:46,338 [macawpublish] INFO - MACAW_SERVICES_HOME is pointing to: C:\macaw-sdk\quickstarts
2017-01-24 15:48:47,200 [macawpublish] WARNING - Tag definition already exists in MDR
2017-01-24 15:48:47,200 [macawpublish] INFO - Service Definitions: ['C:\\macaw-sdk\\quickstarts\\calculator']
2017-01-24 15:48:47,200 [macawpublish] INFO - START: processing service definition - calculator
2017-01-24 15:48:47,200 [macawpublish] INFO - Service docker build directory: C:\macaw-sdk\quickstarts\calculator\impl\dist
2017-01-24 15:48:47,230 [macawpublish] INFO - Remote Build Server (ec2-35-154-114-253.ap-south-1.compute.amazonaws.com) defined. Enabling Remote docker build.
2017-01-24 15:48:47,246 [macawpublish] INFO - Creating (takes couple of seconds to few mins) compressed archive of the docker build artifacts: C:\macaw-sdk\quic
kstarts\calculator\impl\dist\calculator-v1.0.0_dockerbuild_artifacts.tar.gz
2017-01-24 15:48:50,964 [macawpublish] INFO - Launched SSH session to Remote Build Server: ec2-35-154-114-253.ap-south-1.compute.amazonaws.com
Login Succeeded
2017-01-24 15:48:52,198 [macawpublish] INFO - Uploading (takes couple of seconds to few mins) file C:\macaw-sdk\quickstarts\calculator\impl\dist\calculator-v1.0.0_dockerbuild_artifacts.tar.gz via sftp to /tmp/1485301732199_calculator-v1.0.0_dockerbuild_artifacts.tar.gz
2017-01-24 15:50:07,555 [macawpublish] INFO - File uploaded successfully: C:\macaw-sdk\quickstarts\calculator\impl\dist\calculator-v1.0.0_dockerbuild_artifacts.tar.gz
Sending build context to Docker daemon 22.91 MB
Step 1/5 : FROM centos:centos7
 ---> 67591570dd29
Step 2/5 : MAINTAINER macaw.io
 ---> Using cache
 ---> dbd9d2c445fc
Step 3/5 : RUN ln -s /opt/java/bin/java /usr/bin/java && mkdir -p /opt/macaw/calculator
 ---> Using cache
 ---> 75db0a690a56
Step 4/5 : ADD calculator-impl.tar.gz /opt/macaw/calculator
 ---> Using cache
 ---> ca0b464f00cb
Step 5/5 : CMD /opt/macaw/calculator/bin/startup.sh
 ---> Using cache
 ---> 62815e89e505
Successfully built 62815e89e505
2017-01-24 15:50:09,430 [macawpublish] INFO - Docker build successful. Now removing artifact tar.gz file.
2017-01-24 15:50:12,461 [macawpublish] INFO - Docker tagging successful. Now publishing...
The push refers to a repository [172.31.27.188:5000/calculator-v1.0.0]
1cfcbdaa5360: Layer already exists
ce2805f96555: Layer already exists
34e7b85d83e4: Layer already exists
calc-demo: digest: sha256:ae641155533daf96f0a4e56e44b4a03b697e162eb34aa293780a88ca8ed13f3c size: 948
2017-01-24 15:50:13,977 [macawpublish] INFO - Docker Publishing Successful
2017-01-24 15:50:13,993 [macawpublish] INFO - Generating service meta data for publishing to MDR
2017-01-24 15:50:14,007 [macawpublish] INFO - macaw service-info.xml file: C:\macaw-sdk\quickstarts\calculator\impl\src\main\resources\conf\service-info.xml
2017-01-24 15:50:15,055 [macawpublish] INFO - Published to MDR successfully. Tag: calc-demo, Artifact: calculator-v1.0.0, Repo: dev
2017-01-24 15:50:15,055 [macawpublish] INFO - SUCCESS: processed service definition - calculator


Service Repository Status Errors (if any)
------- ----------- ------- ---------------
calculator calculator-v1.0.0:calc-demo success

PS C:\macaw-sdk\tools\macaw-publish-tools\bin>

Login to Macaw Console and Deploy the Calculator Service

• Open the browser and go to https://<your Platform IP or FQDN>
• Login using the credentials. Default credentials are “admin@www.macaw.io” and “admin”
• Browse to the catalogue section and select “onprem MDR/Docker”.
• Locate the blueprint for the calculator service uploaded and click on deploy.

Accessing the Service API

• Go to the Services Section and click on the refresh icon, calculator service as deployed.
• Click on the calculator service and it would reflect the screen below.

• Click on the Browse API and see below the API Browser, where requests can be sent to the service and see the response.

• Select an RPC method from the “Select RPC” drop down box.
• Issue the input that is expected by the calculator service and hit “POST”

• With this, Microservices powered by Macaw Platform end to end have been successfully compiled, published, deployed and tested.

Macaw Eclipse Toolkit

Download and install Eclipse Minimum System Requirements:

• Macaw SDK 0.9.4 or later
• Eclipse Luna. You may choose to download the latest Eclipse installer from https://www.eclipse.org/downloads/

Download software archive for ‘Macaw Eclipse Toolkit’

Download Eclipse Toolkit

Install ‘Macaw Eclipse Toolkit’ software in Eclipse

In your Eclipse workbench, click on Help -> Install New Software… . In the ‘Available Software’ dialog, click on Work with: -> Add… . Provide a name for the software and click on ‘Archive…’ to select the downloaded Macaw Eclipse Toolkit software archive.

Select the ‘Macaw Eclipse Toolkit’ software and press the ‘Next’ button on the dialog. Press the ‘Next’ button on the ‘Install Details’ page. Accept the license on the ‘Review Licenses’ page and press ‘Finish’.

• Accept the license on the ‘Review Licenses’ page and press ‘Finish’.

You will be prompted to trust the software being installed.

Select the certificate; marking the software as trusted; and press the ‘Ok’ button.

After the software is installed, you will be prompted to restart your Eclipse. Press the ‘Yes’ button.

This completes installation of the software.

Configure Macaw Eclipse Toolkit

Select Window -> Preferences -> Macaw.

Select the Java installation directory for ‘JDK Home’. For ‘Macaw Services Home’, select a location where you want the Macaw microservice project artifacts to be generated. Press ‘Apply’ and ‘OK’.

The next step is to configure the toolkit to work with an existing Macaw SDK installation. Select ‘SDK’ preference link under ‘Macaw’ and press ‘Add’ to provide a name and location of the Macaw SDK. The SDK setting will be used to generate Macaw microservices.

You may add different Macaw SDKs. However you need to select one as the default.

After specifying SDK(s), you need to provide repository details where Macaw microservices can be published. Select ‘Publish’ preference link under ‘Macaw’ and press ‘Add’ to provide repository details.

Provide an alias for the repository.
The repository dialog has three tabs namely – MDR, Docker and Build Server.

The ‘MDR’ tab is used to provide the details of the Macaw MDR (Meta Data Repository).

The ‘Docker’ tab is used to provide the details of the Docker repository.

If you do not have docker client installed on your local machine, you can use a Macaw platform instance as remote docker build server. The ‘Build Server’ tab enables you to do that.

Repository Validation

Click on Validate it should success.

Thats it! You are all set to create and publish Macaw microservices.

Create a Service Descriptor

Before we start creating a service descriptor, lets open the ‘Macaw’ Eclipse perspective that prepares your workbench for Macaw project development. Click on the open perspective icon and select ‘Macaw’.

The first step to genarating a microservice project is to create a service definition for the microservice.
The service definition defines the RPCs of the service and the domain entities that the RPCs use.

From the Macaw perspective, navigate to New -> Other… to open the new wizard selection. Select ‘Service Descriptor’ under ‘Macaw’.

The next page allows you to create an empty service descriptor with placeholder. Alternately, you may choose from one of the available templates.

After making your selection, press ‘Next’ button. Provide details of the service module on this page.

Service Namespace: The namespace within which the service components like RPCs and domain entities reside.
Service Name: A unique name for the service.
Service Version: Version of the service in the form of 1.0.0
Review the generated contents on the page that follows and press ‘Finish’. This will open an unsaved editor.

Add or update the contents of the generated service descriptor. Please keep in mind that this file needs to have a valid JSON content.

After editing the file, you need to save the contents of the file. When prompted to save, you may save the file anywhere on your file system.

Generate microservice project

Navigate to New -> Project… to open the project wizard selection. Select ‘Microservice Project’ under ‘Macaw’.

On the next page, select the required configuration for your microservice project.

Select the service desriptor file that you just created in the ‘Input File’. Select ‘Project Language’ to generate the microservice project in Java/Python. Select the build tool of your choice – Ant/Ivy/Maven. Choose the SDK using which you wish to generate the microservice artifacts.

On the next page, provide a name for the package for the generated artifacts and press ->Next–>Next->’Finish’.

The Macaw microservice project is generated at the location that you specified in the preferences (Refer: Configure Macaw Eclipse Toolkit) and imported into your Eclipse workspace as a project.

Update the generated service code.

After putting in the necessary code to have the desired functionality, you need to build the project to generate a publishable artifact. To build the project, select ‘Project’ from the main menu and select ‘Build Project’

The status of the build process can be seen in the ‘Console’ view

Publish microservice project

After building the project, the next step is to publish the project to Macaw MDR and upload the built image to Docker repository. Right click on the project or on any child of the project and select Macaw -> Publish.

In the following dialog, enter the repository that you wish to publish to and provide a unique tag-name for the docker image.

This will trigger the publish activity using the configured default Macaw SDK specified in the preferences (Refer: Configure Macaw Eclipse Toolkit).

Once the publish activity is complete, you will find a success notification.

Deploy Published Service from the Macaw Console

1. 1. Login to the macaw console, Console UI can be accessed at – https://< platform ip >
   2. Provide the requisite credentials and log in.
   3. From the dashboard menu, select Service Manager -> Service Catalog

1. 1. Select the repository that was used to publish and select category “MicroServices
   2. Find your service and click ‘Deploy’

1. 1. On the ‘Deploy Service’ page, select the number of instances required.

1. 1. Select Tag and Click on Deploy

1. 1. Go to API browser and select service ‘whatsmyname’ in this example and click on ADD

1. 1. Select RPC and Provide Input Params and click on Post.Check the output of the RPC call in the output pannel.

This FAQ section includes answers to frequently asked questions as well as helpful troubleshooting instructions.