Saturday, December 25, 2010

"ruby, any messages for me?"

i have successfully consolidated all of my email accounts to gmail. this makes it easy for me to read all my messages from my android phone or other portable internet devices. during the work week i'm rather tied to my email, but it does no good to leave a web browser tab open on the gmail site while i'm teaching class, and it eats up resources unnecessarily. so, i knocked off this little ruby script to check the google xml feed and determine if i have any new mail messages. this simply displays in my conky resource monitor on my desktop, but you can do something fancier, even use this in a more elaborate mail fetching script, if you wish.

01 #!/usr/bin/ruby
02 # gmail.rb
03 # Checks a gmail account and displays the number of unread (new) messages
04 # waiting on the server.
05 # Robert Ritter <rritter@centriq.com>
06 # December 20, 2010
07
08 # Use 'nokogiri' to parse XML.
09 require 'nokogiri'
10 require 'open-uri'
11
12 username = 'not_a_real_user@gmail.com'
13 password = 'not_a_real_password_either'
13
14 tree = Nokogiri.XML(open('https://mail.google.com/mail/feed/atom',
15 :http_basic_authentication=>[username, password]))
16
17 puts tree.xpath('//xmlns:fullcount').first.content
18

until next time!

Monday, December 20, 2010

three wee scripts in powershell 2...

'tis the season of giving. several of my students have been asking for advanced user interface examples in powershell, so here are three scripts (and a module) for them, you, and anyone else who cares to look. they are heavily commented so i'll not go into details, but this is what i got you for christmas...

screenctl.psm1 is a module that contains three useful functions used by the scripts in this set. set-cursor places the cursor exactly where you want it on the screen, so that whatever you print starts displaying there. clear-lines clears a specific number of lines from the screen, rather than the whole thing. combined with set-cursor you can clear just a small section of the screen if you wish. out-wrappedstring is a variation of the text-wrapping function i talked about here.

addusers.ps1 walks you through adding a user account to active directory. it uses the screenctl module to customize the output and the activedirectory module to read and write the database.

get-weather.ps1 provides current conditions and forecast from an internet xml stream. it uses the screenctl module to wrap long forecast strings.

netcfg.ps1 provides a graphical interface to the network information for machines listed in active directory. it uses the active directory module to locate computers.

read the example scripts, improve them, have fun with them this holiday season. it's better than listening to uncle lloyd snore in front of the television, right?

Thursday, August 26, 2010

random passwords in powershell


recently i was teaching my students how to read hundreds of user accounts from a text file and create them in active directory in seconds with a powershell script. to the script we added a nifty function to generate random initial passwords that meet the complexity requirements of windows. here's our function:

9 # new_password
10 # Returns a string.
11 # Generates a random password for the user that meets Windows' complexity
12 # requirements.
13 function new_password {
14 $randomizer = New-Object System.Random
15 $password = [char] $randomizer.next(65,91) # Uppercase letters
16 $password += [char] $randomizer.next(48,58) # Numbers
17 $password += [char] $randomizer.next(91,127) # Lowercase letters
18 # and punctuation
19 foreach ($time in 1..7) {
20 $password += [char] $randomizer.next(32,127)
21 }
22 return $password
23 } # end function new_password
24

don't forget to write the passwords out to a file when you're done so that you can tell users what they are:

.
.
.
62 # Write the username and password to a file.
63 "{0,-19} {1}" -f $samAccountName, $password >> userspass.txt
64 "-" * 30 >> userspass.txt
65

until next time.

Thursday, July 29, 2010

xml 2: and now, a word from our compilers...


last time i demonstrated how to read xml data from a file, looking at the xml support in a few scripting languages. so that we might round out this discussion i will now show how to do the exact same thing with the exact same data using a handful of compiled languages. refer to my previous post to get the xml file and view what the program output should be.

the first language we'll look at is the one of the three that i am least familiar with: c#. despite the fact that i've never used the language outside the classroom and never written a line of “real” production code with it, this was the easiest to use. c# reads and writes much like the scripting languages we looked at last time.


1 // xmlsearch.cs
2
3 // Demonstrates how to parse data in an XML file using C-sharp's XPath
4 // implementation. The file is a collection of books, tech_books.xml.
5
6 using System;
7 using System.Xml;
8
9
10 class XmlSearch {
11
12 static void Main(string[] args) {
13 String separator = "- - - - - - - - - - - - - - - - - - - - - - - - -";
14 XmlDocument tree = new XmlDocument();
15 tree.Load("tech_books.xml");
16
17 // List all the books by title and author from the XML data.
18 XmlNodeList books = tree.SelectNodes("//publisher//book");
19 foreach(XmlNode book in books) {
20 Console.WriteLine("Title: {0}",
21 book.SelectSingleNode("title").InnerText);
22 if(book.SelectSingleNode("subtitle") != null) {
23 Console.WriteLine("Subtitle: {0}",
24 book.SelectSingleNode("subtitle").InnerText);
25 }
26 foreach(XmlNode author in book.SelectNodes("author")){
27 Console.WriteLine("Author: {0}", author.InnerText);
28 }
29 Console.WriteLine(separator);
30 }
31
32 // Get the ISBN and publisher for a book called "Writing Solid Code."
33 Console.WriteLine(separator);
34 String search = "Writing Solid Code";
35 foreach(XmlNode book in books) {
36 if(book.SelectSingleNode("title").InnerText == search) {
37 Console.WriteLine("{0} [{1}] published by {2}", search,
38 book.SelectSingleNode("isbn").InnerText,
39 book.ParentNode.Attributes["name"].Value);
40 }
41 }
42 Console.WriteLine(separator);
43 } // end method Main
44
45 } // end class XmlSearch
46

the next example is in java. while i have worked with java, i've never had to make it read xml before. i was a little disappointed. with factory classes and old-fashioned for loops it's more like c than c#.


1 /* XmlSearch.java
2 **
3 ** Demonstrates how to parse data in an XML file using Java's XPath API.
4 ** The file is a collection of books, tech_books.xml.
5 */
6
7 import javax.xml.parsers.*;
8 import javax.xml.xpath.*;
9 import org.w3c.dom.*;
10
11
12 public class XmlSearch {
13
14 public static void main(String[] args)
15 throws ParserConfigurationException, XPathExpressionException,
16 org.xml.sax.SAXException, java.io.IOException {
17
18 String separator = "- - - - - - - - - - - - - - - - - - - - - - - - -";
19 DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
20 Document tree = factory.newDocumentBuilder().parse("tech_books.xml");
21 XPath xp = XPathFactory.newInstance().newXPath();
22
23 // List all the books by title and author from the XML data.
24 NodeList books = (NodeList)xp.evaluate("//publisher//book", tree,
25 XPathConstants.NODESET);
26 for(int i = 0; i < books.getLength(); i++) {
27 Node book = books.item(i);
28 System.out.printf("Title: %s\n",
29 xp.evaluate("title", book, XPathConstants.STRING));
30 if((xp.evaluate("subtitle", book, XPathConstants.NODE)) != null) {
31 System.out.printf("Subtitle: %s\n",
32 xp.evaluate("subtitle", book, XPathConstants.STRING));
33 }
34 NodeList authors = (NodeList)xp.evaluate("author", book,
35 XPathConstants.NODESET);
36 for(int j = 0; j < authors.getLength(); j++) {
37 System.out.printf("Author: %s\n",
38 authors.item(j).getTextContent());
39 }
40 System.out.println(separator);
41 }
42
43 // Get the ISBN and publisher for a book called "Writing Solid Code."
44 System.out.println(separator);
45 String search = "Writing Solid Code";
46 for(int i = 0; i < books.getLength(); i++) {
47 Node book = books.item(i);
48 if(search.equals(
49 xp.evaluate("title", book, XPathConstants.STRING))) {
50 System.out.printf("%s [%s] published by %s\n", search,
51 xp.evaluate("isbn", book, XPathConstants.STRING),
52 ((Element)book.getParentNode()).getAttribute("name"));
53 }
54 }
55 System.out.println(separator);
56 } // end method main
57
58 } // end class XmlSearch
59

for the sake of completeness, here is the code to do the same thing in c. i'm using libxml2 for its xdom and xpath capabilities. since this is a short example, i have simplified it by excluding all pointer and memory management. don't ever do that in real life. be sure to clean up after yourself: your parent object doesn't work here.


1 /* xmlsearch.c
2 *
3 * Demonstrates how to parse data in an XML file using LIBXML2 and XPath in C.
4 * The file is a collection of books, tech_books.xml.
5 */
6
7 #include <stdio.h>
8 #include <stdlib.h>
9 #include <libxml/parser.h>
10 #include <libxml/xpath.h>
11
12
13 int main(int argc, char **argv) {
14 xmlDocPtr doc;
15 xmlXPathContextPtr tree;
16 xmlXPathObjectPtr result;
17 xmlNodeSetPtr nodeList;
18 xmlNodePtr book, cursor;
19 xmlChar *title, *subtitle, *author, *search, *isbn, *publisher;
20 char *separator = "- - - - - - - - - - - - - - - - - - - - - - - - -";
21 int found = 0;
22
23 doc = xmlParseFile("tech_books.xml");
24 tree = xmlXPathNewContext(doc);
25 result = xmlXPathEvalExpression((xmlChar *)"//publisher//book", tree);
26 nodeList = result->nodesetval;
27
28 // List all the books by title and author from the XML data.
29 for(int i = 0; i < nodeList->nodeNr; i++) {
30 book = nodeList->nodeTab[i];
31 cursor = book->xmlChildrenNode;
32 while(cursor != NULL) {
33 if((!xmlStrcmp(cursor->name, (xmlChar *)"title"))) {
34 title = xmlNodeListGetString(doc, cursor->xmlChildrenNode, 1);
35 printf("Title: %s\n", title);
36 }
37 if((!xmlStrcmp(cursor->name, (xmlChar *)"subtitle"))) {
38 subtitle = xmlNodeListGetString(doc,
39 cursor->xmlChildrenNode, 1);
40 printf("Subtitle: %s\n", subtitle);
41 }
42 if((!xmlStrcmp(cursor->name, (xmlChar *)"author"))) {
43 author = xmlNodeListGetString(doc, cursor->xmlChildrenNode, 1);
44 printf("Author: %s\n", author);
45 }
46 cursor = cursor->next;
47 }
48 puts(separator);
49 }
50
51 // Get the ISBN and publisher for a book called "Writing Solid Code."
52 puts(separator);
53 search = (xmlChar *)"Writing Solid Code";
54 result = xmlXPathEvalExpression((xmlChar *)"//publisher", tree);
55 nodeList = result->nodesetval;
56
57 for(int i = 0; i < nodeList->nodeNr; i++) {
58 cursor = nodeList->nodeTab[i];
59 publisher = xmlGetProp(cursor, (xmlChar *)"name");
60 book = cursor->xmlChildrenNode;
61 book = book->next;
62 while(book != NULL) {
63 cursor = book->xmlChildrenNode;
64
65 while(cursor != NULL) {
66 if((!xmlStrcmp(cursor->name, (xmlChar *)"title"))) {
67 title = xmlNodeListGetString(doc,
68 cursor->xmlChildrenNode, 1);
69 }
70 if((!xmlStrcmp(cursor->name, (xmlChar *)"isbn"))) {
71 isbn = xmlNodeListGetString(doc,
72 cursor->xmlChildrenNode, 1);
73 }
74 cursor = cursor->next;
75 }
76 if((!xmlStrcmp(title, search))) {
77 found = 1;
78 break;
79 }
80 book = book->next;
81 }
82 if(found) {
83 printf("%s [%s] published by %s\n", title, isbn, publisher);
84 break;
85 }
86 }
87 puts(separator);
88
89 return 0;
90 } // end method main
91

Sunday, July 25, 2010

reading xml 1: powershell, python and ruby, oh my!


i was working on a little project that required me to read an xml file from the internet. since i like to have some useful examples to refer to when i'm doing something that i don't often do (and so don't remember how to) i decided to dig into the three scripting languages that i'm most likely to use and learn how to work with xml data.

you can grab the sample xml file here: tech_books.xml. i recommend that you look it over so you can see what the scripts are doing. we'll be writing scripts to list all the book titles and authors, then search for a specific title and fetch its isbn and its parent publisher's name attribute. all three should provide the same output, which you can see below:

Title: The Revolutionary Guide to Assembly Language
Author: Vitaly Maljugin
Author: Jacov Izrailevich
Author: A. Sopin
Author: S. Lavin
- - - - - - - - - - - - - - - - - - - - - - - - -
Title: Writing Solid Code
Subtitle: Microsoft's Techniques for Developing Bug-Free C Programs
Author: Steve Maguire
- - - - - - - - - - - - - - - - - - - - - - - - -
Title: Windows PowerShell Scripting Guide
Subtitle: Automating Administration of Windows Vista and Windows Server 2008
Author: Ed Wilson
- - - - - - - - - - - - - - - - - - - - - - - - -
Title: Practical C Programming
Author: Steve Oualline
- - - - - - - - - - - - - - - - - - - - - - - - -
Title: Programming Python
Author: Mark Lutz
- - - - - - - - - - - - - - - - - - - - - - - - -
Title: The Ruby Way, Second Edition
Author: Hal Fulton
- - - - - - - - - - - - - - - - - - - - - - - - -
Title: LaTeX: A Document Preparation System
Author: Leslie Lamport
- - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - -
Writing Solid Code [ISBN 1-556-15551-4] published by Microsoft Press
- - - - - - - - - - - - - - - - - - - - - - - - -

the first language i turned to is python, an old friend of mine. reading through the xml document object model api made my head swim, but with a little research i came across what used to be a third-party module, elementtree, that has now been incorporated into the core python installation. (this is one of the reasons that i love open source software.)

elementtree uses parent.findall(child) to fetch an array of all nodes of type child under parent. the parent.find(child) method returns only a single element object. parent.findtext(child) returns the child node data. it is similar to parent.find(child).text: the former returns an empty string if the element is not found, the latter raises an exception. element.get(attribute) returns a node's attribute value.


1 # xmlsearch.py
2
3 import xml.etree.ElementTree
4
5
6 tree = xml.etree.ElementTree.ElementTree()
7 tree.parse('tech_books.xml')
8
9 # List all the books by title and author from the XML data.
10 books = tree.findall('publisher/book')
11 for book in books:
12 print "Title: %s" % book.findtext('title')
13 if book.findtext('subtitle'):
14 print "Subtitle: %s" % book.findtext('subtitle')
15 authors = book.findall('author')
16 for author in authors:
17 print "Author: %s" % author.text
18 print '- ' * 25
19
20 # Get the ISBN and publisher for a book called "Writing Solid Code."
21 print '- ' * 25
22 search = 'Writing Solid Code'
23 publishers = tree.findall('publisher')
24 for publisher in publishers:
25 books = publisher.findall('book')
26 for book in books:
27 if book.findtext('title') == search:
28 print "%s [ISBN %s] published by %s" % (search,
29 book.findtext('isbn'), publisher.get('name'))
30 print '- ' * 25
31

next i investigated xml under powershell. i was certain that it would be no more difficult than python, and what i found was that basic xml searches worked in a very similar way.

powershell has xml processing built-in, so no modules have to be included in the source file to make it work. it uses parent.SelectNodes(child) to get an array of all nodes called child under the node parent. the parent.SelectNode(child) method returns only the first node child that is found under parent. note the plural SelectNodes and singular SelectNode. element.GetAttribute(attribute) returns a node's attribute value.


1 # xmlsearch.ps1
2
3 [xml] $tree = Get-Content tech_books.xml
4
5 # List all the books by title and author from the XML data.
6 $books = $tree.SelectNodes('//publisher//book')
7 foreach ($book in $books) {
8 "Title: {0}" -f $book.title
9 if ($book.subtitle) {
10 "Subtitle: {0}" -f $book.subtitle
11 }
12 $authors = $book.SelectNodes('author')
13 foreach ($author in $authors) {
14 "Author: {0}" -f $author.'#text'
15 }
16 '- ' * 25
17 }
18
19 # Get the ISBN and publisher for a book called "Writing Solid Code."
20 '- ' * 25
21 $search = 'Writing Solid Code'
22 $publishers = $tree.SelectNodes('//publisher')
23 foreach ($publisher in $publishers) {
24 $books = $publisher.SelectNodes('book')
22 foreach ($book in $books) {
23 if ($book.title -eq $search) {
24 "{0} [ISBN {1}] published by {2}" -f $search, $book.isbn,
25 $publisher.GetAttribute('name')
26 }
25 }
26 }
27 '- ' * 25
28

finally i dusted off ruby, a language that i haven't really touched in more than a year, to see what it would do. again, i wasn't impressed with the general xml documentation, but google searches led me to a ruby gem called nokogiri. i installed it and was very pleased with the result.

nokogiri uses parent.xpath(child) to get an array of child elements, parent.at_xpath(child) to get a single element, and element.attr(attribute) to get a node's attribute value. one nifty property lacking in the other two languages is element.parent, which returns the parent node of the current element. it saves us from having to define multiple “books” arrays in this example.


1 # xmlsearch.rb
2
3 require 'rubygems'
4 require 'nokogiri'
5
6
7 tree = Nokogiri.XML(open('tech_books.xml'))
8
9 # List all the books by title and author from the XML data.
10 books = tree.xpath('//publisher//book')
11 books.each do |book|
12 puts "Title: %s" % book.at_xpath('title').content
13 if book.at_xpath('subtitle')
14 puts "Subtitle: %s" % book.at_xpath('subtitle').content
15 end
16 authors = book.xpath('author')
17 authors.each do |author|
18 puts "Author: %s" % author.content
19 end
20 puts '- ' * 25
21 end
22
23 # Get the ISBN and publisher for a book called "Writing Solid Code."
24 puts '- ' * 25
25 search = 'Writing Solid Code'
26 books.each do |book|
27 if book.at_xpath('title').content == search
28 puts "%s [ISBN %s] published by %s" % [search,
29 book.at_xpath('isbn').content, book.parent.attr('name')]
29 end
30 end
31 puts '- ' * 25
32

so after doing a little research i find that it's not really too difficult to read xml in my favorite scripting languages, and now i have some nice little examples to look at the next time i need a refresher. until next time.

Wednesday, July 14, 2010

powershell: my 3 favorite functions (2 of 2)


you may have noticed that when you print a really, really long string in powershell, the text is wrapped at the edge of the console. if your screen is 80 characters wide, your string gets wrapped at 80 characters. you may also have noticed that this wrapping isn't very intelligent. your string will be wrapped at exactly 80 characters, even if that means that the line gets broken in the middle of a word. just look through the powershell help to see what i mean.

textwrap

python has a nifty little module called textwrap that wraps a string the smart way. i don't know a lot about it—i only use the parts i need. it is not my intent to re-write the python module for powershell, only to borrow the concept with my own textwrap function. the idea is to take a long string and a maximum width in characters and return a string formatted so that each line does not exceed the width. whole words should be preserved at the end of each line so that the text is readable. because this is a more complex function, we will build it incrementally and talk about it along the way.


C:\> function textwrap ($longstring, $width) {
>>> $word_list = -split $longstring
>>> $out_strings = @("")
>>> $i = 0
>>>

the simple way to begin is to take the long string and convert into an array of words using the string split() method or, if you're using powershell 2, the split operator as i did here. we're going to convert the string to an array of strings, each no longer than the desired width. we'll store this in $out_strings. notice we initialized the array with an empty string. that way we can write to $out_strings[0] without getting errors about non-existent array elements. we'll start indexing our array at zero with our good buddy $i.


>>> foreach ($word in $word_list) {
>>> if ($out_strings[$i].Length + $word.Length -gt $width) {
>>> $i++
>>> $out_strings += ""
>>> }
>>> $out_strings[$i] += $word
>>>

what we want to do is concatenate each word to the end of the string in our array as long as that wouldn't make the string longer than our maximum width. we use the foreach loop to fetch each word from the list. in the loop we test to see if we can safely add the word to the end of the string in the first array element. once we've added as many words to the string as we can, we increment the array index and append a new empty string to the array. then we add the next word to the end of the new string.


>>> if ($out_strings[$i].Length + 1 -le $width) {
>>> $out_strings[$i] += " "
>>> }
>>> }
>>> return ($out_strings -join "`n")
>>> }
>>>

before we leave the foreach loop we need to add a space to the end of the word we just concatenated onto our string. that is, as long as it doesn't put us above our width. the if statement above adds a space if there is room to do so. finally, we return our array of strings as a single string with the join operator. we separate them with a newline character so that the new string prints on multiple lines. given a long string stored in $ls a sample run of this function might look like this:


C:\> "-" * 25
-------------------------
C:\> textwrap $ls 25
This is a really long
string. It is wrapped in
a way that doesn't chop
up words.
C:\>

the finished function appears below. i've added code so that it can even deal with hyphenated words and do the right thing. i'll let you examine that to see how it works. until next time.


33
34 # textwrap:
35 # expects a string and an int
36 # returns a string
37 # Intelligently wraps a long string to a maximum width.
38
39 function textwrap ($long_string, $width) {
40 $word_list = -split $long_string
41 $out_strings = @("")
42 $i = 0
43
44 foreach ($word in $word_list) {
45 # If adding the word would make the final string too long, start a
46 # new string:
47 if ($out_strings[$i].Length + $word.Length -gt $width) {
48 # If the word is hyphenated, try just the first part:
49 if (([regex] "\w+\-\w+").IsMatch($word)) {
50 if ($out_strings[$i].Length + 1 +
51 $word.Split("-")[0].Length -le $width) {
52 $out_strings[$i] += $word.Split("-")[0]
53 $out_strings[$i] += "-"
54 $word = $word.SubString($word.IndexOf("-") + 1)
55 }
56 }
57 $i++
58 $out_strings += ""
59 }
60 $out_strings[$i] += $word
61
62 # Add a space if it doesn't make the string too long.
63 if ($out_strings[$i].Length + 1 -le $width) {
64 $out_strings[$i] += " "
65 }
66 }
67
68 return ($out_strings -join "`n")
69 } # end function textwrap
70

Monday, July 12, 2010

powershell: my 3 favorite functions (1 of 2)


i have every intention of continuing our lessons in x86 assembler and nasm, but right now there is something else on my mind. let's pause our look at assembler for a little while and discuss powershell.

powershell is a very powerful scripting language for managing windows hosts. in fact, somewhere else i've gone on record saying, “if you administer a windows network, you must learn powershell.” i meant that.

powershell is very flexible and much easier to customize than dos batch files of yore. the powershell profile, reminiscent of the .profile used in the unix bourne shell, allows you to collect your personal preferences and code where they will always be available when you run a script. to that end there are three functions that, for me and my profile, are “must-haves.” i'll share them with you here.

pause

the first is the simple dos pause command. i'm not sure why powershell doesn't have such a useful command. it has proven its worth through decades of batch files. nevertheless, the powershell team has left it out. no matter. we can write our own with hardly any effort at all.

5
6
7 # pause:
8 # accepts a string
9 # Prints a message on the screen, then waits until a key is pressed before
10 # proceeding. If a string is not passed to the function the default string
11 # will be used. Simulates the "pause" command in DOS.
12
13 function pause ($msg = "Press any key to continue...") {
14 $msg
15 $Host.UI.RawUI.ReadKey("NoEcho, IncludeKeyDown") | Out-Null
16 } # end function pause
17

this function will accept a string to print as your pause message, but will default to the old dos standard “press any key” in absence of one. it uses the console's ui.rawui.readkey method to capture a raw key press, and redirects the output to the out-null cmdlet to avoid echoing the key's character on screen. a sample run of this function should look like this:

C:\> pause
Press any key to continue...
C:\> pause "I can't do that, Dave."
I can't do that, Dave.
C:\>

test

the second function is test, borrowed from the linux bourne-again shell, or bash. the test command takes an expression and determines whether the result is true or false. now, the bash command isn't really intended to be user-friendly: the result of the test is stored in the program's exit code. this is useful in a shell script, but i like to use test to help me debug scripts and check that my expressions are giving me the results i expect, so i've expanded it a little for powershell.

18
19 # test:
20 # expects an expression
21 # Accepts any valid expression and evalutes it for truth. Prints a message
22 # indicating that the expression evaluated true or false.
23
24 function test ($expr) {
25 if ($expr) {
26 Write-Host -ForegroundColor Green "True"
27 }
28 else {
29 Write-Host -ForegroundColor Red "False"
30 }
31 } # end function test
32

we can use this function from the interactive prompt to determine if some logical expression does what we think it does. a sample run of the function would look like this:

C:\> test (3 -gt 4)
False
C:\> test ("Bill" -is [string])
True
C:\> test (42 - 6 -lt 30 -or 56 + 9 -gt 70)
False
C:\>

now we can easily test even a complex expression to see whether it should be true or false when we use it in a script.

be sure to check back next time for the coolest of my three favorite functions.

Friday, April 30, 2010

assembler tutorial: 2 - goodbye, mr. chips


and bade them curt 'hello,' and then 'good-bye.'


for our second heavily-annotated example, we'll modify the 'hello world' program a bit. beyond simply saying "goodbye" this new app will allow the user to provide a name on the command line to which we may bid adieu. if no argument is provided, the program will simply say goodbye to mr. chips. this time we will be assembling a dos com file.

in dosbox, move to the mycode directory and launch editv to create a new file called goodbye.asm.

comment, comment, comment. note that this file will not require a linker.


1 ; goodbye.asm
2 ; This variation on the 'Hello World' program expands upon the original
3 ; a little by accepting a name from the command line and printing that
4 ; in the message. If no parameter is supplied, the program will use a
5 ; default value. This one is written as a COM file, so it doesn't
6 ; require linking.
7 ; To assemble: nasm -o goodbye.com goodbye.asm
8 ;
9 ; Robert Ritter
10 ; 25 Apr 2010
11


we begin with a directive for the assembler. org is not a machine instruction, but rather a note to the assembler so that it can configure all of the segment addresses for you. in a dos com file the entire program fits into a single 64kb segment, so loading up all those segment registers seems rather silly. org 100h tells nasm that this is a com file, so all segments are at exactly the same address, and the program starts at offset 100h. why this offset? well the first 256 bytes (0h through ffh) makes up the program segment prefix, or psp. byte 100h is the first place real code can be loaded. since i'm defining my data at the top of my source file, i'm putting a jmp (unconditional jump) at this address to tell the system to skip right to the good stuff. jmp works like the much-maligned goto statement in other languages: it transfers execution to the code found at the given label. we're going to let the program flow jump to the label called 'Start' and we'll catch up in a bit.


12 ; ----------------------------------------------------------------------
13 org 100h
14 ; We set up a COM file by defining the address of the program location
15 ; in memory, which will always be 100h. Then we jump to the start of
16 ; the code block.
17 ;
18 jmp Start
19


you've seen data before, and you will recognize db from our last program. one new thing here is the equ directive. this creates a constant. data defined with db may be modified during program execution, but data defined with equ cannot. any attempt to change the value stored in endMsgLen in this program will cause the assembler to balk with the message that the label has been redefined.

another new thing is the use of the dollar sign outside the quotation marks. what does that mean? well, we're going to be copying strings into a buffer and we'll need to tell the cpu exactly how many bytes to copy. it's easy to find the size of the beginMsg string: we subtract its address from the address of defaultMsg.
defaultMsg - beginMsg
will give us the length of beginMsg. remember that labels are just aliases for memory addresses. we can use the same technique to find the length of defaultMsg. to find the length of the last string, endMsg, we subtract endMsg from $. the dollar sign in line 31 means "this byte right here." so endMsgLen will contain the difference between endMsgLen and endMsg. that's pretty cool.


20 ; ----------------------------------------------------------------------
21 section .data
22 ; DOS COM files don't use segmented memory. The whole program fits
23 ; into a single 64KB block, so there's no need to worry about segments
24 ; at all. The assembler still expects to find defined data and code
25 ; sections, though, and it helps us to organize our source if we keep
26 ; things compartmentalized like this.
27 ;
28 beginMsg db 'Goodbye, '
29 defaultMsg db 'Mr. Chips'
30 endMsg db '!', 0dh, 0ah, '$'
31 endMsgLen equ $ - endMsg
32


the bss section (so named for historic reasons) contains data that is not initialized to a specific value; at least, no value that we care about. here we're creating a working buffer to which we may copy the elements of our final string before we send it to standard output.


33 ; ----------------------------------------------------------------------
34 section .bss
35 ; This section contains unintialized storage space. We allocate space
36 ; here for data that we won't have until runtime. COM files don't
37 ; require an explicit STACK section. The assembler will take care of
38 ; the stack for us.
39 ;
40 fullMsg resb 1024 ; This is the message we will print.
41 ; We'll assemble it from parts and
42 ; copy each part into this memory area.
43


the rep movsb instructions copy a sequence of bytes from one place in memory to another. the number of bytes that get copied is found in the cx register. so what we're doing here is concatenating strings and storing the result in fullMsg. first strings first...

 
44 ; ----------------------------------------------------------------------
45 section .code
46
47 Start:
48 ; First we'll copy the beginning of the message, 'Goodbye,' to our
49 ; allocated memory. The number of bytes to copy (the length of our
50 ; data) goes into CX.
51 mov cx, defaultMsg - beginMsg
52 ; The address of the data goes into SI (think Source Index) and the
53 ; address of the allocated memory into DI (as in Destination Index.)
54 mov si, beginMsg
55 mov di, fullMsg
56 rep movsb ; REP MOVSB copies CX bytes from SI to DI.
57 ; DI is automatically incremented.
58


remember that psp? the first 128 bytes (00h through 7fh) is full of stuff that we're really not interested in, but the second 128 bytes (80h through ffh) contains information from the command line that we used to run our program. since we want to get a name from the parameter list on the command line, we want to read this part of the psp. byte 80h tells us how long the parameter list is, so if it's zero (the program was run without any parameters) we'll say goodbye to the default name; otherwise we'll read the parameter list and take our name from there.

throughout our sojourn in assembler we've been working with memory addresses. the programming savvy among you may have said to yourself, "ah, these are pointers." most of what we work with in assembler is addresses, or pointers. if you learned in computer programming class that pointers were hard, then you learned them incorrectly; but that's a rant for another post. suffice it to say that pointers are the way to manipulate data in assembler. however, sometimes we need to get at the data in a memory location directly. on line 71 we need to compare zero to the value at address 80h, not the address itself. nasm makes this pretty easy: we use square brackets around an address to access the value inside. the instruction in line 69 means, "copy the value stored at address 80h into the cl register." there, you've just dereferenced a pointer. no big deal.

this section contains a couple of logic branches using the cmp operator to compare two values, a jz operator to jump if zero to a particular label, and an unconditional jmp instruction to skip parts of the program that won't be used if a name was given on the command line. remember that labels are just memory addresses in assembler.


59 ; Next we'll copy the command line parameter into our allocated memory.
60 ; When we start a program, DOS creates a data structure for it called
61 ; the PSP (Program Segment Prefix) that loads ahead of it in the first
62 ; 256 (100h) bytes of memory. (This is why the COM file has to point to
63 ; address 100h to start.) The first 128 bytes of the PSP is "stuff," so
64 ; we won't worry about that. The last half of the PSP contains the
65 ; parameter string. Byte 80h contains the length of the string and the
66 ; remaining bytes contain the parameter string terminated by a carriage
67 ; return (0dh.)
68 xor cx, cx ; Set CX to zero.
69 mov cl, [80h] ; Put the parameter length
70 ; into CL.
71 cmp cl, 0 ; Test CL to see if it's zero.
72 jz NoParam ; If CL contains zero jump to
73 ; another part of the program.
74 ; If the JZ (Jump if Zero) wasn't executed, then the user ran the
75 ; program with a command line parameter. CX now contains the number of
76 ; bytes in the string, but the first byte is always a space, so we'll
77 ; decrement CX and start copying the string from byte 82h. DI already
78 ; points to the end of the last thing we copied to memory.
79 dec cx
80 mov si, 82h
81 rep movsb
82 jmp FinishString ; Skip the NoParam part since
83 ; there was a parameter.
84
85 NoParam:
86 ; No parameter was given on the command line. We'll use the default
87 ; goodbye message.
88 mov cx, endMsg - defaultMsg
89 mov si, defaultMsg
90 rep movsb
91
92 FinishString:
93 ; Now we copy the last part of the message to memory.
94 mov cx, endMsgLen
95 mov si, endMsg
96 rep movsb
97
98 ; Use the DOS service call to print the string that is in memory.
99 mov dx, fullMsg
100 mov ah, 09h
101 int 21h
102
103 ; Exit with no error code.
104 mov ax, 4c00h
105 int 21h
106


save the file and exit editv. assemble the program directly into a com file:
nasm -o goodbye.com goodbye.asm
run the program with and without parameters.

now that was a pretty sophisticated program. you fetched data from the command line and used a condition to branch to a specific part of your program, much like if..then logic found in so-called high-level languages, and you used a default parameter if one wasn't provided. your skills are progressing nicely, padawan. you're training is almost complete.

Monday, April 12, 2010

assembler tutorial: 1 - hello, world!


but thunder interrupted all their fears


now that you're ready to begin writing we'll dig right in. here is our first heavily-annotated program, hello.exe.



open up your dosbox and change to the mycode directory.
cd mycode
run editv with a new file, hello.asm.
editv hello.asm
i strongly advise you to turn on line numbering. you can do this by pressing ctrl-o followed by b, or you can just select the options menu with alt-o, scroll down to line numbers and press enter.

we begin with comments. if you've ever read a book on programming it has probably emphasized the need for good comments. in assembler comments are even more important because the language syntax is so terse. a comment begins with a semi-colon and continues to the end of the line of text.


1 ; hello.asm
2 ; Demonstrates how to write an assembly program for DOS with NASM using
3 ; the ubiquitous 'Hello World' string. To create an EXE file we'll
4 ; first assemble an OBJ file then link it. I'm using the public domain
5 ; linker WarpLink.
6 ; To assemble: nasm -f obj hello.asm
7 ; To link: warplink hello.obj
8 ;
9 ; Robert Ritter <rritter@centriq.com>
10 ; 12 Apr 2010
11


remember that dos addresses memory in segments. the first thing we'll need to do is reserve some memory for these segments. since this program doesn't work with a lot of data our data segment is pretty small. it defines a label, message, that will be the memory address of the first byte of the message that we're going to print out. the db (define byte) operator identifies a sequence of bytes that make up our data. there is also a dw (define word) for 16-bit values, and dd (define double word) for 32-bit values. most of the time you'll probably just treat data as a sequence of bytes, so you'll likely use db more than the others.

you may have noticed the characters that follow the obvious string, "Hello World." if we want to advance our output to the next line we must insert a newline character. this is like pressing the enter key on a keyboard. in high-level languages like c we use a string like "\n" to represent a newline, but in dos this is actually a two-byte sequence: 0dh (carriage return) and 0ah (linefeed.) the dollar sign character is a terminator that marks the end of the string. not all strings must be terminated with a dollar sign, but the dos printing service that we're going to use requires it.

notice that the characters that make up a string are enclosed in quotes. double or single quotes, it makes no difference. those characters outside the quotes are treated as literal bytes.


12 ; ----------------------------------------------------------------------
13 segment data
14 ; DOS EXE files use segmented memory which allows them to address more
15 ; than 64KB at a time. Here we define the data segment to store the
16 ; message that we're going to print on the screen.
17 ;
18 message db 'Hello World', 0dh, 0ah, '$'
19


the next thing that we want to do is reserve some memory for our stack segment. the resb (reserve byte) operator is used to set aside an uninitialized piece of memory of a given size. there is also a resw (reserve word) for 16-bit values and resd (reserve double word) for 32-bit values. we're going to allocate a 64-byte hunk'o'ram for the stack and set the label stackTop to point to the address immediately following the stack. for more info on how the stack works, see my previous post.


20 ; ----------------------------------------------------------------------
21 segment stack stack
22 ; The stack is used as temporary storage for values during the
23 ; program's execution. Sometimes we use it in our code, and sometimes
24 ; DOS uses it, especially when we call DOS interrupts. We'll set up a
25 ; small but serviceable stack for this program since we're going to be
26 ; calling on DOS services.
27 ;
28 resb 64
29 stackTop ; The label 'stackTop' is the address of the end (top)
30 ; of the stack. We'll need this to initialize the
31 ; stack pointer in the CPU.
32


the code segment is where the cool stuff happens. remember that a dos exe file may have more than one code segment to get around that pesky 64kb barrier we discussed last time. though multiple code segments are allowed, only one can be the actual entry point of our program. this is defined with a special label, ..start. note that i used a colon at the end of this label. a label may end with a colon, but this is not required. you may find code examples that are pretty inconsistent on the use of colons in labels. even examples in the official nasm documentation waffle a little on this. personally, i choose to use a colon when the label refers to a block of code, and to forgo the colon when the label refers to data. remember, though, that to the assembler they're all just addresses.

we're giving the mov operator a real workout here. the instruction
mov dest, src
tells the assembler to copy the data at src into dest. yes, it goes right to left, but you get used to it pretty quickly. in this instance we're loading segment addresses into their respective cpu registers. since we can't copy immediate data directly into a segment register, we'll use ax for temporary storage.


33 ; ----------------------------------------------------------------------
34 segment code
35 ; The code segment is where our program actually does stuff. Executable
36 ; instructions go here.
37 ;
38 ..start:
39 ; First we need to do some housekeeping. Our program needs to know at
40 ; what addresses its segments can be found. The Intel CPU contains some
41 ; special registers just to hold this information, so we'll load them
42 ; up now. Since we can't put addresses directly into these registers,
43 ; we'll copy them to the AX general purpose register first.
44 mov ax, data
45 mov ds, ax ; DS: data segment register
46 mov ax, stack
47 mov ss, ax ; SS: stack segment register
48 mov sp, stackTop ; SP: stack pointer register
49


now we're going to call on dos to print our message on the screen. dos and the system bios have several services that they offer to our programs. these are accessed by triggering an interrupt with the int instruction. each service has its own requirements, so we need to look up the particular service we want in our handy dos developer's guide in order to properly use it. the dos service we're using here is service 09h of the general purpose interrupt 21h. to use it we place the address of a dollar-sign-terminated string into register dx, place the service id 09h into register ah, then call interrupt 21h.


50 ; We're going to use a DOS service to write a string to the screen.
51 ; The documentation for this service says that we have to terminate the
52 ; string we want to print with a dollar sign (see how we did this in
53 ; the data segment above) and we must put the address of the string
54 ; into the DX register and call the service. DOS interrupt 21h provides
55 ; all kinds of cool services. To use it we place the service ID in
56 ; register AH and call INT 21h.
57 mov dx, message
58 mov ah, 09h
59 int 21h
60


finally we exit the program. we'll use service 4ch of dos interrupt 21h. if you have a specific exit code (for example, to signal an error) you place it into register al. just as before, we put the service id into ah and call the interrupt. since we have no error condition we'll do a clean exit. here we load al and ah at the same time by putting 4c00h into ax.


61 ; We also use INT 21h to exit our program. The exit function is 4Ch,
62 ; which goes into AH. The exit code that is used to report errors back
63 ; to the operating system goes into AL. We'll just load both at the
64 ; same time, then call INT 21h.
65 mov ax, 4c00h
66 int 21h
67


you have just written a program in assembly language. save the file and exit editv. assemble the file with nasm:
nasm -f obj hello.asm
this will create an object file suitable for linking into a dos exe. link the file with warplink:
warplink hello.obj
this creates the file hello.exe. notice that i included these instructions in the comments at the top of the source file. this is useful if you come back to the program at a later time and want to make changes. now run your program and bask in the warmth of the knowledge that you have made this cpu do your explicit bidding. a little bit more of this and you'll be ready for live minions.

next time we'll pass command-line parameters into our program, and we'll shake things up a bit with the dos com file format.

Monday, April 5, 2010

assembler tutorial: intro 2 of 2


the lovers moved to flee from heaven's tears


warning: this post contains frequent references to explicit hex, and may be inappropriate for readers under the age of 11h

last time i said, "before we begin in earnest, two things." here comes thing number two.



second, a word about how assembler works. you are no doubt aware that your computer has long-term storage (disks) and short-term storage (random access memory, or ram.) if we use an office allegory to describe a computer, we might say that the disks are like the filing cabinets in the back room: they can hold lots and lots of stuff, and are generally pretty well organized, but inconvenient. constantly going to them to fetch new work or to put something away would be a chore, so we tend to use them only when we need to grab something we plan to use soon or to put something away when we won't be using it for a good long while. ram is like the in/out trays on my desk: i can stack all kinds of stuff there (though much less than i can put in the filing cabinets) and my work is quickly and easily accessible. the cpu is like my desktop, where all the work actually happens. to do some work i have to take it from the trays and move it to the desktop, and to clear the desk for some other task i need to move what's on the desktop back to the trays. so where in the cpu do we store this really temporary stuff while working on it?

registers

cpus have built-in memory storage spaces called registers. in the intel x86 architecture, 16-bit general purpose registers go by the names ax, bx, cx and dx. each 16-bit register can be broken into two parts, a high-order byte and a low-order byte. for register ax, these would be called ah and al respectively. the specific meanings of high- and low-order aren't too important right now, and the topic delves deep into ancient religious wars of cpu design, but suffice it to say that putting the 16-bit word c725h into register ax will load c7h into ah and 25h into al.

in modern cpus each 16-bit register is only half of one of the 32-bit registers, which bear the names eax, ebx, ecx and edx. there are other special purpose registers that we'll talk about as we move along, but you get the idea.

so writing an assembly language program is like shuffling paperwork around. you copy data into a register, you tell the cpu to process it, then you do something (or nothing, if you wish) with the result. here is a simple set of instructions that you'll see frequently in assembler. we'll talk about what it does next time.

1 ; These instructions are commonly found in DOS programs.
2 mov ax, 4c00h
3 int 21h


segments

another thing you must know is how dos accesses memory. to mov (copy) data to or from ram you need an address. since dos uses 16-bit registers, the largest address it can work with is 16 bits long, so dos can address up to 65,536 (64k) bytes of ram. that's it. a long time ago 64k was a lot. remember all the great programs we ran on the commodore 64? but as consumers demanded more from their applications 64k became a barrier. dos handles this by viewing memory as a series of segments, each 64kb in size. a program can contain many segments of code and data so long as none of them exceeds 64kb.

to address memory, then, we need two registers: a special segment register for the segment address and a normal general purpose register for the offset within that segment. if the ds register, which points to a data segment, contains 24a0h and we mov 0fh into register dx, then ds:dx refers to the 16th byte of that segment, written as 24a0:000fh. if we later load ds with 4110h we'll find that ds:dx now points to 4110:000fh. it's not too complicated, but it's up to the programmer to keep track of which segment he's using at any point in time. fortunately you need not know the exact addresses of your segments (dos actually determines that at runtime, so there is no way you could know as you're writing your source.) in assembler we use labels, friendly names to refer to addresses. so you may see code like the following to initialize the data segment register:

4 ; Load the DS register with the address of the data segment.
5 mov ax, data ; "data" is the address of our
6 mov ds, ax ; data segment


stack

finally, there is the stack. this is a handy little place in ram to put things temporarily, such as when you want to pass data from one procedure to another. we push data onto the stack to store it, and we pop data off of the stack to retrieve it. the stack is like that little cart with the clean plates at the head of a buffet line. the most recently cleaned plates are warm, damp and on the top of the stack, and the ones that have been there awhile and are much drier are at the bottom. when you take a plate off the top, you're taking the one that was most recently placed on the stack.

data stacks work the same way. the topmost item is the most recently pushed data, the oldest data is at the bottom. data is always popped in reverse order from how it was pushed onto the stack.

of course, we all know what happens when you put too many plates in a stack on one of those carts. bad, loud things happen. if we were to overfill our stack segment in our program, we could overwrite some other segment, or worse, some other program's segment. this could also lead to bad, loud things, so the intel cpu does a funny thing when it sets up the stack: it fills it backward, from the top down. you need to see this to get it…

let's say that you decide to create a stack segment for your program that is only four bytes long (don't use such a small stack in real life.) the ss register (stack segment) will contain the address of the beginning, or bottom of the stack. the first byte would be at ss:0h, the second at ss:1h, the third at ss:2h and the fourth at ss:3h. the stack pointer (another register called sp) will point to the top of the stack, 4h.

"wait!" you cry. "4h isn't in the stack segment, because it's only four bytes long!" you're right. what's at the address that sp is pointing to right now? we don't know for sure. "isn't that dangerous?" you ask. perhaps, but wait until you see how the thing comes off.

when we push a byte onto the stack, sp is first decremented, so now it points to 3h. then the pushed data is copied to 3h. see? everything is okay, because we don't actually write to 4h, so we don't corrupt some other program's stuff. when we push a second byte, sp is decremented and the new data is written to 2h. when we pop the data off of the stack, the data that sp points to at 2h is copied and sp is incremented to point to 3h. but what if we try to push more than four bytes onto the stack? well, consider that when the fourth byte is pushed sp has been decremented four times, so it now points to 0h. any attempt to decrement sp again will set the overflow flag in the flags register, and dos will crash your program with a stack overflow error. your program valiantly falls on its own sword to keep from doing bad things to other programs. of course, there is nothing to keep you from popping the data at 4h before you've pushed anything onto the stack. just expect really bad things to happen when you try to use that unknown value. it's best to not go there.

why use the stack if it's so much potential trouble? remember that programs love it, more than programmers love their buffets. sometimes procedures pass parameters by the stack so that they can do things. after branching to another line of execution the stack can act like a trail of breadcrumbs, helping your code to wend its way back to where it came from. even if you never consciously use it, the services you request through dos or bios interrupts will use the stack. buck up, young padawan: you can't escape your destiny.

now then, you're all set to write your first program in assembly language…

Tuesday, March 30, 2010

assembler tutorial: intro 1 of 2

assembled 'neath a dark, foreboding sky


as promised, a little dos and linux assembly language tutorial in several verses for your consideration. mostly dos though, for that is where my dark past lies, but a little linux is good for the soul (and the understanding.) before we begin in earnest, two things:


first, assemble (huh) your 'a' team, the tools you will need to code along with me. since i work on mac, linux and vista platforms i find great comfort in the familiarity of a singular environment across all machines that i (might) happen to be writing code on at the moment. for this, i use dosbox, a great little dos + x86 emulator. it may have been primarily designed for running classic games like wolfenstein or ega trek, but it makes for a pretty decent development platform, too.

to write your code you'll need a text editor. alas, a good dos text editor for writing assembler is only a pipe dream (this used to not be the case, but i can't find my old tools anymore.) editv, however, does provide must-have line numbering and is one of the easiest editors to use. i still prefer vim, but the dos version of it is really lame and i don't intend to teach vi editing in this blog.

we'll be writing code specifically to be assembled by nasm, the netwide assembler. Hello World Text there is no such thing as "standard assembler syntax," so every assembler has its own idiosyncrasies. one of the reasons i'm writing these tutorials is to teach myself nasm: i actually learned on tasm and worked on masm in the bronze age of computing. the concepts are the same and linguistic differences are often minor, so it's nothing to break a sweat over.

some of the programs we write will require a linker. i'm using the public domain warplink for this. and finally, since most of these utilities are packaged as .zip files, we'll snag the darling of dos bbses the world over, pkzip (the second-finest piece of dos shareware ever written.)

install dosbox and create a folder in your home directory to keep your work in. i called mine dos. edit the dosbox configuration file by adding a few lines after the [autoexec] section at the very end. mine automatically mounts my dos directory as drive c and adds several directories to my path variable, like this:

[autoexec]
# Lines in this section will be run at startup.
@echo off
mount c ~/dos
c:
set PATH=%PATH%;C:\NASM;C:\WARPLINK;C:\EDITV;C:\PKZIP;C:\BIN

once you've done this you can start dosbox (which should drop you into your c drive) and use the dos command md to create the directories nasm, warplink, editv, pkzip, bin, and mycode. run the command dir and ensure that eight directories exist (this number includes the . and the .. directories.)

copy pk250dos.exe to the dos/PKZIP directory and run the executable inside dosbox. you will find that when you modify the dos filesystem from your host operating system outside dosbox, those changes don't appear when you list the directory inside dosbox. dos likes to cache the directory lists to speed things along, so if it is unaware of changes you made from outside the environment it pretends that they aren't there (they really are.) to force the changes to appear you can clear the directory cache with the rescan command.

copy the rest of the programs listed above to their respective directories and, in dos, run rescan and unzip each of them like this example:

C:\EDITV>pkunzip editv41u.zip

when all this is done, close and reopen dosbox (you can type 'exit' at the dos prompt to close) and verify that you can run all these programs from the root directory of drive c by just typing their names: editv, nasm, and warplink. if you run into any dosbox trouble, refer to this little article. if everything works as expected, your path is set up correctly, your team is assembled (huh) and you're… almost… ready to begin…

Sunday, March 21, 2010

small packages, part 2

a while back i said that i like small programs that waste neither space nor cpu cycles. from the file on stupid computer tricks comes the extreme case: the smallest possible linux program that actually does something.

the exercise documented in this article (updated for 2009) is purely academic, but it demonstrates some uncommon ingenuity, and i think that's cool. and this recent blog post is moving along the same lines: programming in c with no cumbersome libraries. not really practical for large projects where the standard library accounts for a negligible part of program size and the benefits far outweigh the drawbacks, but still this is an interesting effort. it also caused me to dig out my old assembly language books. i think i'm going to write some tutorials on intel assembler over the next few months. we'll look at making programs for dos and linux.

aww, c'mon! it'll be fun!