Maintaining a Private NPM registry for your Organization with Sinopia–Private NPM server


In this post, we will take a quick dig at setting up our own private Node Package Registry with Sinopia.

“sinopia is a private/caching npm repository server”

Maintaining a private NPM for your organization/team is very helpful when you would like to share the code only with them and not the entire universe. You can develop and maintain all your organization specific projects and their components as node packages that can be reused.

If there are multiple teams working, each of them can upload their own reusable node modules to the private NPM and other teams can use them. For example a File upload component or an organization specific HTML5 boilerplate and so on. This way, when you need something all you have to do is run npm install my-h5bp , which will download the boilerplate for you.

So, let us get started.

Official Approach

The folks at NPM have given us a process on how we can replicate the NPM privately.

Screen Shot 2014-09-21 at 10.16.33 amYou can check out this blog post to achieve the same. But this process is quite complex.

Another enterprise solution for setting up your private NPM is NPME – NPM Enterprise. You can find more info on that here : npm Enterprise. A quick video

Other Approaches

There are couple of other solutions like Kappa and node-reggie, which enable you to set up a private NPM with few quick steps.

But some how I was drawn to sinopia for its quick installation, setup and the ease of use. Sinopia uses a file system to manage the registry. The best part about Sinopia is, it does not sync the public registry by default, but it will cache the packages only when downloaded for the first time. You can save some space on your server (with the fast growing public registry and all).

Setup and Configure Sinopia

Setting up Sinopia as mentioned earlier is pretty easy. You need to have Node installed on the machine, where you are setting up the private NPM. Open terminal/prompt and  CD  to the root folder

Windows
Mac/*nix

> cd /
$ cd ~

Inside the root directory, create a new directory named sinopia

mkdir sinopia && cd sinopia

Next, we will install sinopia globally. Run

npm install -g sinopia

(use sudo if needed)

Now, we will start the server run

sinopia

If you are launching the server for the first time, you will be asked to create a config file. Select Yes.

Note : If you launch the server from a different folder and if the folder does not have the config file you will be asked this question again. (I do not know why!) So make sure you launch the server from the same folder, the one we have cd ‘ed into earlier.

Screen Shot 2014-09-21 at 11.09.19 am

Now, when you navigate to http://localhost:4873/  you will see a message like

1

Web interface is a work-in-progress right now, so it is disabled by default. If you want to play with it, you can enable it in the config file.

To enable the web interface, we need to tweak the config file. From the current folder, open theconfig.yaml file. Scroll to a section named the web and set enabled to true.

Screen Shot 2014-09-21 at 11.00.55 amKill the sinopia server and relaunch it. Now navigate to  http://localhost:4873  and you should see the web interface

Screen Shot 2014-09-21 at 11.02.35 amSweet right!!

[Important Step]

Now, from a client machine where the private NPM will be accessed from, you need to set the NPM registry URL pointing to our Private NPM server and not the public registry.

If your server and client are same, you can run

npm set registry http://localhost:4873

And if your server and client are on different machines, you need to run

npm set registry http://path.to.your.server:4873

In your config.yaml you can configure the host and port too. Navigate to the Advanced section and you can updated the config like

Screen Shot 2014-09-21 at 11.16.22 amKill sinopia and restart it. Now your web URL will be  http://localhost:2772. Reverting the port back to 4873, we will continue.

Setup User

By default there is an Admin user setup, which you can find under the users section in the config.yaml.

Screen Shot 2014-09-21 at 11.22.19 amThis is the default admin account and the sha1 encrypted password. I am not sure what this password is, so I will reset to admin$123.

For that, open a new terminal/prompt. Run node  then execute

crypto.createHash(‘sha1’).update(‘admin$123’).digest(‘hex’) and paste it back to the config file

Screen Shot 2014-09-21 at 11.29.11 amSave the config file after pasting the generated hash, restart sinopia and run

Screen Shot 2014-09-21 at 11.34.59 amThe password would be admin$123.

You are logged in!!

To restrict access to the private registry, you can run

npm set always-auth true

This way you can secure all your private packages (more on conditional securing in a moment).

Now, a new client who would like to access the registry would need to create an account by running

npm adduser –registry http://localhost:4873/

Provide user details and bam!! You have a newly authenticated account for your private registry

Screen Shot 2014-09-21 at 11.48.24 am Now you will see a new file created inside the sinopia folder

sinopia folder

Shell

sinopia  tree

|– config.yaml

`– htpasswd

0 directories, 2 files

The htpasswd file consists of the authentication data. It is better not to mess with this file.

Downloading packages

Now that we have everything setup, we will download our first package. Create a new folder named privateProj and open a new terminal/prompt inside this folder

Now run

npm init

to start a new node project. Next, we will install diskDB a node package for managing JSON files.  If you do notice your sinopia folder, you will see that there are only 2 files. The config and the htpasswd file. Now once we start pulling the packages, a new folder named storage will be created. You can configure the storage folder path in config.yml. Screen Shot 2014-09-21 at 11.59.23 am

From inside the privateProj run

npm install diskdb –save

And you should see

Screen Shot 2014-09-21 at 12.01.14 pmDo remember, diskdb is not in our private registry. Sinopia has fetched this package from the public repo. Now, if you see the sinopia folder, you will see

Screen Shot 2014-09-21 at 12.02.58 pmIf the public registry is offline or not available for some reason, sinopia will fetch the packages from the cache.  You can configure this behaviour in the uplinks section.

Screen Shot 2014-09-21 at 12.06.12 pmNow that we have diskdb installed in our privateProj, we will build a simple app and publish it to our private registry.

Build & Publish a Node module

You can check out my post Write your own Node Modules on how to create your own node modules and publish it.

Do note that as long as your registry URL is pointed to the private repo, the publish command would not save the package to the public registry.

Now, to build a sample node module/reusable component, we will create a folder named dband a file named app.js inside the privateProj folder.

Update the app.js as below

app.js

JavaScript

var db = require(‘diskdb’);

db = db.connect(‘db’, [‘fruits’]);

var fruits = [‘Apple’, ‘Mango’, ‘Orange’];

db.fruits.save(fruits);

var printFruits = function() {

console.log(db.fruits.find());

}

// To test the app locally

// printFruits(); // << uncomment this

// and run

// $ node app.js

exports.printFruits = printFruits;

Now, our app is ready. We would like to share this app with our team, so they can reuse theprintFruits().

For that we would be publishing this app to our private registry. Run

npm publish

and you should see something like

Screen Shot 2014-09-21 at 12.20.02 pmThe user arvind does not have access to publish to our private registry. For this to work, we need to provide access. Open config.yaml and scroll down to the packages section and add the user name to allow_publish.Screen Shot 2014-09-21 at 12.23.18 pmSave the file and restart the sinopia server. Now when you publish you should see

Screen Shot 2014-09-21 at 12.25.16 pmAnd when you navigate to  http://localhost:4873 you should see

Screen Shot 2014-09-21 at 12.26.08 pm

Now, if you see the sinopia folder,

Screen Shot 2014-09-21 at 12.27.11 pm

Simple and easy right!!

Now, if someone wants to use your package all they need to do is run

npm install privateProj –save

The same way one would from the public registry and you should see

Screen Shot 2014-09-21 at 12.29.20 pm

And then in your project, you can directly use

myApp.js

JavaScript

var pp = require(‘privateProj’);

pp.printFruits();

And run

node myApp.js

If you want to maintain a different “namespace” for your private packages, you can do so by prefixing your packages with your organization name or your project name and update theconfig.yaml like

Screen Shot 2014-09-21 at 12.56.19 pm

You can also setup a different storage for your private repos too.

That is all you need to setup your private NPM and manage packages.

Note : If you followed the above post on a local machine and wanted to revert your npm registry to the public one, execute

npm config set registry https://registry.npmjs.org

or

npm config set registry http://registry.npmjs.org


How to host Sinopia in IIS on Windows?

package.json

{
  "name": "iisnode-sinopia",
  "version": "1.0.0",
  "description": "Hosts sinopia in iisnode",
  "main": "start.js",
  "dependencies": {
    "sinopia": "^1.3.1"
  }
}

image

start.js

process.argv.push('-l', 'unix:' + process.env.PORT);
require('./node_modules/sinopia/lib/cli.js');

web.config

<configuration> <system.webServer> <!-- indicates that the start.js file is a node.js application to be handled by the iisnode module --> <handlers> <add name="iisnode" path="start.js" verb="*" modules="iisnode" /> </handlers> <rewrite> <rules> <!-- iisnode folder is where iisnode stores it's logs. These should never be rewritten --> <rule name="iisnode" stopProcessing="true"> <match url="iisnode*"/> <action type="None"/> </rule> <!-- Rewrite all other urls in order for sinopia to handle these --> <rule name="sinopia"> <match url="/*" /> <action type="Rewrite" url="start.js" /> </rule> </rules> </rewrite> <!-- exclude node_modules directory and subdirectories from serving by IIS since these are implementation details of node.js applications --> <security> <requestFiltering> <hiddenSegments> <add segment="node_modules" /> </hiddenSegments> </requestFiltering> </security> </system.webServer> </configuration>

—————————————————————————————————————————

Sinopia Commands

npm publish (to publish package) or npm publish –tag <tag>

npm unpublish –force (to unpublish / delete package)

npm install / update (Installing packages (npm install, npm upgrade, etc.)

npm dist-tag add <package>@<version>

npm adduser {newuser}

npm search

Useful Links

1. http://resolvethis.com/how-to-create-a-new-sinopia-user/

2. http://resolvethis.com/how-to-setup-a-private-npm-repository-with-sinopia/

3. https://www.npmjs.com/package/sinopia

4. http://perrymitchell.net/article/private-npm-repository-with-sinopia/

 

Happy Coding Smile

angular- Dynamic window height with ng-style


Recently, I was working on one nw.js (Node-webkit) related project for creating windows / desktop application using node/nw.js. I had to write window resize event and based on that I needed to show vertical scrollbar.

in short something like – where height and scrollbar will adapt window resize automatically upon resize.

image

angular code

app.directive(‘winResize’, function ($window) {
    return function (scope, element, attr) {

        var w = angular.element($window);
        scope.$watch(function () {
            return {
                ‘h’: window.innerHeight,
                ‘w’: window.innerWidth
            };
        }, function (newValue, oldValue) {
            console.log(newValue, oldValue);
            scope.windowHeight = newValue.h;
            scope.windowWidth = newValue.w;

            scope.resizeWithOffset = function (offsetH) {
                scope.$eval(attr.notifier);
                return {
                    ‘height’: (newValue.h – offsetH) + ‘px’                   
                };
            };

        }, true);

        w.bind(‘resize’, function () {
            scope.$apply();
        });
    }
});

HTML code

{{content}}

window.height: {{windowHeight}}

window.width: {{windowWidth}}

Hope this helps someone in need…!! Happy Coding Smile

How-to: node.js Logger–winston setup / configuration


I am working on node.js based REST API development and I am using this very popular npm middleware called “Winston” for logging errors, exceptions and even info & debug log too.

Why we need it?

Now you must be wondering why we need it since we already have console.log to check logs while development, BUT what happens when we decide to go live? How to access the logs? How to create different log levels?

Well, WINSTON is your answer… following is the setup and configuration.

Manual Logging

If you want to log only specific things that you want to track, there are easy libraries that let you do so. Winston is a very popular one. You can log your output to multiple transports at the same time (e.g: file, console…). It’s also supported by 3rd parties cloud services that will be happy to host those outputs for you (more about that next). 

Here is a simple snippet of how to configure winston and log it to both your console and a file: 

 

var winston = require('winston');

//log your output to a file also

winston.add(winston.transports.File, { filename: 'somefile.log' });

//log some outputs

winston.log('info', 'Hello distributed log files!');

winston.error('Who let the dogs out?!');

How if you want dedicated log files to have all log written to it? which is idle for production anyways –  Here is my setup

– Create folder “helpers” in your project & create “logger.js” in that folder.

– logger.js code

image

– Now in your server.js or app.js (your main node startup file) add this middleware declaration

var logger = require(‘./helpers/logger.js’);

– That’s it…. You are all set…verify by logging any simple information like –

logger.info(‘this is information’);

logger.debug(‘this is debug’);

logger.warn(‘this is warning’);

logger.error(‘this is error’);

Thanks to Winston development team & of course sweet node.js…

Happy Coding Smile

iisnode installation & configuration


Recently I am working on Node.js REST API Development & wanted to deploy that on IIS. So here is my configuration steps –

Hardware:

Windows 8 x64 bit

IIS 8.0

Software:

Download IISNode from https://github.com/tjanczuk/iisnode

image

Go to – https://github.com/azure/iisnode/wiki/iisnode-releases

image

Verify

open iis (Start->run->inetmgr)  & Go to Modules

image

You should see following iisnode modules now registered in iis

image

Then I created sample IISNode + Nodejs application. & hosted under defaults site in IIS as below. You should see iisnode modules in that new site as well, else it wont work.

image

Sample code web.config

<configuration>
  <system.webServer>
    <!– indicates that the hello.js file is a node.js application
    to be handled by the iisnode module –>
    <handlers>
      <add name=”iisnode” path=”server.js” verb=”*” modules=”iisnode” />
    </handlers>
    <rewrite>
      <rules>
        <rule name=”api”>
          <match url=”api/*” />
          <action type=”Rewrite” url=”server.js” />
        </rule>
      </rules>
    </rewrite>
    <directoryBrowse enabled=”false” />
    <iisnode
      devErrorsEnabled=”true”
      debuggingEnabled=”true”
      loggingEnabled=”false”
      debuggerPathSegment=”debug”
      nodeProcessCommandLine=”C:\Program Files (x86)\nodejs\node.exe”
      promoteServerVars =”APPL_MD_PATH”>
      <!– NOTE: promote server vars is for middleware : iis-baseUrl which is in middlewares folder–>
    </iisnode>
  </system.webServer>

  <system.web>
    <compilation debug=”true” />
  </system.web>
</configuration>


   

server.js code

var http = require(‘http’);
var port = process.env.port || 1337;
http.createServer(function (req, res) {
res.writeHead(200, { ‘Content-Type’: ‘text/plain’ });
res.end(‘Hello, world! [helloworld sample; iisnode version is ‘ + process.env.IISNODE_VERSION + ‘, node version is ‘ + process.version + ‘]’);
}).listen(port);

Hope this helps… Happy Coding Smile

How-to with Node.jS & file system


1. Reading from a file
Problem:

You want to read a file from the file system and display its contents on the terminal screen.

Solution:
var fs = require('fs');
fs.readFile('data.txt', function (err, data) {  if (err)    throw err;  if (data)    console.log(data.toString('utf8'));
});
Discussion:

First, we include the file system core module using module inclusion as written: var fs = require(‘fs’);  allowing us to utilize all the features of the fs module using our variable we defined as fs. The next line states our input file ‘data.txt’. The next part of this line will attach our anonymous callback function function (err, data) to the actual file read operation readFile.

Our anonymous callback function accepts any errors which occur as  err, and the resulting file data as a binary buffer array  data. Within the anonymous function, we first check for any potential errors and throw the error as an exception to be left to be caught by the calling process as if (err)  throw err; . The subsequent lines check to see if the data is not null, and then converts the data to a UTF-8 encoded string to be written to the console screen if (data) console.log(data.toString(‘utf8′));

It is important to note that this solution uses the asynchronous file read operation to prevent blocking. For a synchronous, or blocking file read operation, reference the Node.js documentation for readFileSync()

var fs = require('fs');
fs.writeFile('data2.txt', 'Hello, World!', function (err) {
     if (err)
       throw err;
});
Discussion:

First, we include the file system core module using module inclusion as written: var fs = require(‘fs’);  allowing us to utilize all the features of the fs module using our variable we defined as fs.  The next line states our input file ‘data2.txt’ along with a string of the text we want to write to the file ‘Hello, World!’. The next part of this line will attach our anonymous callback function function (err) to the actual file write operation writeFile(). Our anonymous callback function accepts any errors which occur as  err. Within the anonymous function, we also check for any potential errors and throw the error as an exception to be left to be caught by the calling process as if (err)  throw err; .
It is important to note that this solution uses the asynchronous file write operation to prevent blocking. For a synchronous, or blocking file write operation, reference the Node.js documentation for writeFileSync()

3. Working with directories
Problem:

You want to read through a list of all of the files in a given directory location and display each file’s name.

Solution:
var fs = require('fs');
fs.readdir('.', function (err, files) {
 if (err)
    throw err;
 for (var index in files) {
    console.log(files[index]);
 }
 });
Discussion:

First, we include the file system core module using module inclusion as written: var fs = require(‘fs’);  allowing us to utilize all the features of the fs module using our variable we defined as fs.  The next line states our input directory ‘.’ (the current working directory). The next part of this line will attach our anonymous callback function function (err, files) to the actual directory read operation readdir(). Our anonymous callback function accepts any errors which occur as  err and an array of filenames stored in files.

Within the anonymous function, we first check for any potential errors and throw the error as an exception to be left to be caught by the calling process as if (err)  throw err;. Next we iterate over each item in the filename array and display each filename on the console screen.

It is important to note that this solution uses the asynchronous directory read operation to prevent blocking. For a synchronous, or blocking directory reading operation, reference the Node.js documentation for readdirSync()

4. Traversing the file system
Problem:

You want to recursively read through a list of all of the files in a given directory location and display each file’s name or recursively traverse the file if it is a directory.

Solution:
 var fs = require('fs');
 var traverseFileSystem = function (currentPath) {
    console.log(currentPath);
    var files = fs.readdirSync(currentPath);
    for (var i in files) {
       var currentFile = currentPath + '/' + files[i];
       var stats = fs.statSync(currentFile);
       if (stats.isFile()) {
       console.log(currentFile);
       }
      else if (stats.isDirectory()) {
             traverseFileSystem(currentFile);
           }
     }
   };
  traverseFileSystem('..');
Discussion:

First, we include the file system core module using module inclusion as written: var fs = require(‘fs’);  allowing us to utilize all the features of the fs module using our variable we defined as fs.  The next line defines our custom function traverseFileSystem which accepts one parameter currentPath which contains the current directory path the function is traversing.

The first line of this function simply displays the current directory’s name. Next we synchronously return a list of files contained in the current directory (this is done synchronously to guarantee we have a list of files before we try to traverse them). Next we iterate over each item in the filename array. From here we create a variable to store our fully-qualified file path and obtain information about that file using a synchronous file stats call  statSync() (this function is described in more depth in the next example).

The function then checks if the current file is a file or a directory, and either displays the file’s name if it is a file or recurses by re-invoking the traverseFileSystem function using the new current directory path if it is a directory. The last line of the code invokes the  traverseFileSystem function on ‘..’ which is a simple way to reference the parent directory of the current working directory.

It is important to note that this solution uses the synchronous versions of some of the asynchronous methods used in previous examples in order to guarantee return values before proceeding with the code’s execution. This was done in the sake of simplicity in illustration of this solution and can be ported into an asynchronous version with little difficulty

5. Getting/setting file attributes
Problem:

You want to get certain attributes of a given file such as whether or not it is a file or a directory, and you want to be able to set certain attributes such as the file’s name or the file’s permissions.

Solution:
 var fs = require('fs');
 fs.stat('data.txt', function (err, stats) {
  if (err)
     throw err;
  if (stats.isFile()) {
      console.log('It\'s a file!');
  }
  if (stats.isDirectory()) {
    console.log('It\'s a directory!');
  }
  for (var i in stats) {
     if ('function' !== typeof stats[i])
     console.log(i + '\t= ' + stats[i]);
   }
  });
 fs.rename('data2.txt', 'data2_new.txt', function (err) {
  if (err)
    throw err;
  console.log('Renamed!');
 });
 fs.chmod('data3.txt', '0777', function (err) {
   if (err)
      throw err;
   console.log('File permissions changed!');
 });
Discussion:

First, we include the file system core module using module inclusion as written: var fs = require(‘fs’);  allowing us to utilize all the features of the fs module using our variable we defined as fs.

The next line obtains information about a given file ‘data.txt’ using an asynchronous file stats call  stat(). On this same line we specify our anonymous callback function function (err, stats) which takes two parameters: any errors which occur as  err and a stats object of the given file as stats. Within the anonymous function, we first check for any potential errors and throw the error as an exception to be left to be caught by the calling process as if (err)  throw err;.

We then display on the console screen whether the given file is a file or a directory. After this, we step through each non-function member of the stats object and display what that member field’s name is and what value it contains (this will include detailed file information such as creation date, modified date, file size, etc).

The next section of the code executes the asynchronous rename() function to rename the file ‘data2.txt’ to ‘data2_new.txt’. In this same line, we are passing the anonymous callback function function (err) which, as before, passes any potential errors that might occur as its only parameter err.

We then use the same technique to check for any errors and then display ‘Renamed!’ if the file was successfully renamed. In the last section of the code we execute the asynchronous chmod() function to change the file permissions of our input file ‘data3.txt’ to ’0777′ (or all read-write-execute). In this same line, we are passing the anonymous callback function function (err) which, as before, passes any potential errors that might occur as its only parameter err. We then use the same technique to check for any errors and then display ‘File permissions changed!’ if the file permissions were successfully changed.

6. Watching a file for changes
Problem:

You want to be notified of any changes that occur within a given directory either to the directory’s contents or directory itself.

Solution:

This solution requires a special module called “inotify.” Assuming you have the node package manager installed, you can install inotify using: npm install inotify The code below must then reference the install path:

 var fs = require('fs');
 var Inotify = require('/usr/local/node/node_modules/inotify').Inotify;
 var inotify = new Inotify();
 var watcher = {
 path: '.',
 watch_for: Inotify.IN_OPEN | Inotify.IN_CLOSE,
 callback: function (event) {
 var file = event.name ? event.name : '';
 var mask = event.mask;
 if (file != '' && file.indexOf('.') !== 0) {
    if (mask & Inotify.IN_OPEN) {
    console.log(file + ' was opened ');
 }
 else if (mask & Inotify.IN_CLOSE) {
    console.log(file + ' was closed ');
 }
 }
 }
 };
inotify.addWatch(watcher);
Discussion:

First, we include the file system core module using module inclusion as written: var fs = require(‘fs’);  allowing us to utilize all the features of the fs module using our variable we defined as fs. We then include the inotify modulevar inotify = require(‘/usr/local/node/node_modules/inotify’).Inotify;  (note that this path should be whatever path you installed the inotify module to). On the next line, we are instantiating the inotify object new Inotify(); which we will use for watching for file changes.

The next few lines are all for defining our  watcher object with the following properties: path: ‘.’ (the path to the file we want to watch—in this case, we are watching the current working directory), watch_for: Inotify.IN_OPEN | Inotify.IN_CLOSE (specifying that we will only be firing change events when a file is opened or closed), and lastly, callback: function (event) {  … } (defines our callback function that will be fired whenever the above two events are emitted). In the first line of our callback, we are getting the filename from the event object var file = event.name ? event.name : ”;  (if the filename is null, we’ll set the variable to an empty string).

The next line is just to make a “shorthand” variable for the event type mask var mask = event.mask; .  The next conditional statement will filter out empty filenames and filenames that start with a “.” (which sometimes indicate a temporary file being used to write a file). The next couple of statements if (mask & Inotify.IN_OPEN) {  … } and then after  else if (mask & Inotify.IN_CLOSE) {  … }  will display whether there was a file opened or closed under the given directory location.

Lastly is a very important line of code where we add our watcher object that we just defined to the inotify object using inotify.addWatch(watcher); (note that without this very line, our watcher object is essentially useless).

For more info – https://nodejs.org/api/fs.html

Happy Coding Smile

Everything about Node.js (How-to-start)


Tutorials

Developer Sites

Videos

Screencasts

Books

Courses

Blogs

Podcasts

JavaScript resources

Node.js Modules

Other

Happy Coding Smile