It appears that you're running an Ad-Blocker. This site is monetized by Advertising and by User Donations; we ask that if you find this site helpful that you whitelist us in your Ad-Blocker, or make a Donation to help aid in operating costs.

UBB.Sitemaps

What Is Covered

Summary
About the Modification
Additional Files
     • Supplied robots.txt Instructions
Caching
Demonstration
Install Instructions
Usage Instructions
Settings
     • Configuration Settings
     • Database Connection Information
     • Cache & Optimization Options
Human Readable XML Styling
Submissions
UBB.threads Version & URLs
Changelog
Troubleshooting
     • Invalid Date Error
Frequently Asked Questions
     • What does purchasing a license entitle me to?
     • What are the security aspects?
     • What is a Sitemap?
Useful Links
Thank You


 

Summary


This page represents the full documentation for the UBB.Sitemaps script by VNC Web Services.


 

About the Modification


Requirements: UBB.threads 7.x (not compatible with UBB.threads 6.x or below)
Current Version: v0.10
License Information: This script is not freeware, it requires a license; by purchasing a license you're granted usage on 1 (one) UBB.threads installation. Any additional installations require an additional license. Updates are available for 1 year after initial purchase; after this timeframe, to aid in development costs, you must renew your access to updates (see the purchase link below for further information). You cannot freely provide this script to others without written permission from the author.
Pricing: $50 (Purchase Here)
Support: Support is available on the UBB.Dev thread [7.x] UBB.Sitemaps.

UBB.Sitemaps is a script that we created due to some reported search engine "discovery" issues with the way that URLs where initially written in the UBB.threads product that some users where reporting. Not to mention that a lot of users had moved from previous or competing products and wanted to see their URLs immediately discovered in the Google, Bing, Yahoo, or other search
engine(s).

This script aids in the "discovery" portion of search engine crawling; instead of waiting and hoping that your URL's get crawled, an XML Sitemap alerts Google (or any crawlers which allow sitemaps) as to what URLs exist on your forums, so they know what to crawl based on what you tell them.

Most search engines will take your feed once a week or so and run it through to determine new URLs, on a Google Webmaster Tools account, you can browse exactly how many URLs have been included, as well as when the last time they have hit your site.

Please note however, that Google will still crawl your site and try to perform its own discovery, so please do not use a robots.txt entry to block it (as it must be able to crawl the content provided in the sitemap in order to populate your entries).

The configuration options allow you to specify which type of sitemap you wish to use and whether you wish to plugin threaded or flat view. The sitemap type and view type can both be set either by hard coding the appropriate variable or via a URL string.


 

File Listing


Your install archive will contain several files, namely:
• cache/sitemap/ - A blank directory to show the proper nesting for the default cache directory.
• css/sitemap.css - A CSS Stylesheet for XML Feeds.
• readme.txt - A brief readme which gives basic tips on getting started, and a link here to the full documentation.
• robots.txt - A listing of instructions for crawlers which support robots.txt; see below for a listing of the supplied instructions.
• sitemap.xsl - An XSL Style.
• ubb.sitemap.php - The UBB.Sitemaps script itself.

 

Supplied robots.txt Instructions


Though not required, we recommend applying the items in the robots.txt file in the attached folder so that Google and other search engines do not attempt to crawl areas in which they commonly get trapped. Please be sure that if you utilize the items in the robots.txt file that you update the path to reflect that of your forums.

Supplied robots.txt Instructions
User-agent: *
Disallow: /forum/templates/
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php/ubb/calendar.html
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php/ubb/showday.html
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php/ubb/online.html
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php/ubb/search.html
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php/ubb/faq.html
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy.html
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies.html
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php/ubb/markallread.html
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser
Disallow: /forum/ubbthreads.php/ubb/newuser.html
Sitemap: http://www.example.com/ubb.sitemap.php?type=1&se=1



 

Caching


In v0.6 we added several caching and optimization areas to the script; these will help alleviate load issues from multiple clients grabbing your map. Please note that at times there may be some strict server configurations which won’t work well with these options, your mileage may vary.

The cache option will build a copy of the output and store it for a defined period of time, any user accessing the file during this period will be served the cached file vs the live file (with the exception of the initial visiting user who will be served the live file.

The exception header option will allow you to set a date "in the future" when the content of the file "should" change, we default this to 24 hours so the client should cache this file on their side for this time frame.

To utilize caching you will need a cache/sitemap/ folder (from the script instance) that is chmodded 777.

The SHA1 hash of the test URL (for the cache directory) is e0f94273f1b777a695d0f53fd7acc4c111942675.


 

Demonstration


The UBB.Sitemaps script offers several sitemap generation styles, these are:
UBB.Sitemaps - ASP Sitemap
UBB.Sitemaps - Link Sitemap
UBB.Sitemaps - RSS Sitemap
UBB.Sitemaps - Sitemap Index
UBB.Sitemaps - Text Sitemap
UBB.Sitemaps - XML Sitemap

In v0.9 we introduced the ability to style your XML and Test output pages; which make them more human readable (and functional in the case of the XSL style):
UBB.Sitemaps - Style (CSS)
UBB.Sitemaps - Style (XSL)


 

Install Instructions


Open the ubb.sitemap.php file in a plain text editor (such as notepad or EditPlus) and adjust the variables for your environment; all of the settings include comments which give information on the setting as well as the defaults.

Note
The default configuration settings will generate an XML sitemap which will be fine for any search engines that support them (Google and Bing most notably).


After customizing the script settings you'll need to upload the script to your web root; in order for Google or other search engines to read that the sitemap belongs to your domain, the sitemap generator must be in the html document root (http://www.example.com/).


 

Usage Instructions


We've built the sitemap generator to be quite flexible, there are several options and parameters that you can customize for your live feed of your sitemap for numerous uses scenarios as:
• Sitemap Types: 0/Test, 1/XML, 2/Text, 3/html, 4/RSS, 5/ASP, 6/SitemapIndex (Default: 6/SitemapIndex)
• Use SE Friendly or Search Engine Optimized URLs (Default: 1/Yes)
• Use ShowFlat or ShowThreaded. (Default: 0/ShowFlat)
• Offsetting links to break up large forum sitemaps into manageable chunks. (Valid values are from 0-50000; Default: 10000)

And all variables can output either "as default" by directly accessing ubb.sitemap.php or by defining a parameter in the URL, such as:
Google would be happiest with: http://www.example.com/ubb.sitemap.php?type=6&se=1 or http://www.example.com/ubb.sitemap.php?type=1&se=1 (That's type=6 for a Sitemap Index (or type=1 for XML), and se=1 for SEO/Spider Friendly URL's. We recommend that everyone just default to using a SitemapIndex as it future-proof's your site (as a Sitemap can only have up to 50k links; however, loading the ceiling maximum of data, to be loaded at once, can cause strain on your server. Using a SitemapIndex will break your Sitemap up into more manageable chunks).

In the past, yahoo was more comfortable with RSS feeds, but recently they've also adapted to work well with Google's XML sitemaps.

As this script pulls data live from the database, it is unwise to link directly to the various output options unless it's being fed into search engines as there is a potential for a lot data output. We recommend always using the caching option.

To test to ensure it is working before providing the feed URL's to search engines, please load your sitemap generator with the following parameters (as the preview uses a small limit it shouldn't take longer than a couple of seconds to execute):
http://www.example.com/ubb.sitemap.php?type=0&se=1

The sitemap index only works with XML sitemap feeds based off of the sitemaps.org standard; as these are only valid under XML sitemaps they display the output according to that standard only. To grab the sitemap index (should you have the need to utilize the offset option) please access it as:
http://www.example.com/ubb.sitemap.php?type=6&se=1

If you are _NOT_ going to use a SitemapIndex and want to load the entire dataset at once, you'll want to change the "offsets" option to not utilize them, otherwise it'll load the amount of data up to the default offset chunk size (of 10k).


 

Settings


 

Configuration Settings


     $conf["connection"] = 1; // 0 = Set DB Info (below); 1 = Pull data from UBB.threads Config. (Default: 1)
     $conf["folder"] = "forum/"; // The folder that your UBB.threads is located in, Example: http://www.example.com/forum/ would be "forum/" (with trailing slash) (Default: "forum/")
     $conf["frequency"] = 4; // 0 = Never, 1 = Always, 2 = Hourly, 3 = Daily, 4 = Weekly, 5 = Monthly, 6 = Yearly (Default: 4)
     $conf["friendly"] = 1; // Use SE Friendly/SEO Links? 0 = No, 1 = Yes (Default: 1)
     $conf["offsets"] = 1; // 0 = No, 1 = Yes (Default: 1)
     $conf["style"]["xml"] = 0; // 0 = No, 1 = CSS (sitemap.css), 2 = XSL (sitemap.xsl) (Default: 0)
     $conf["style"]["name"] = "sitemap.xsl"; // The name of the sitemap style; an example css/sitemap.css and sitemap.xsl are supplied. (Default: "sitemap.xsl")
     $conf["title"] = "Community Name"; // You can set this to change the name of your feed. By leaving this by default (or blank) and selecting to use data from your config (see $conf["connection"]) it will use your community name.
     $conf["type"] = 1; // 0 = Test, 1 = XML, 2 = Text, 3 = HTML, 4 = RSS, 5 = ASP, 6 = Sitemap Index (Default: 1)
     $conf["version"] = 3; // UBB.threads Version, 0 = v7.0 - v7.2.2, 1 = v7.3 - v7.4.2, 2 = v7.5 Series, 3 = v7.6 Series (Default: 3)
     $conf["view"] = 1; // What view would you like to use? 1 = ShowFlat, 2 = ShowThreaded (Default: 1)

 

Database Connection Information


     $conf["db"]["password"] = "sql_password"; // This should be the password to your database.
     $conf["db"]["prefix"] = "ubbt_"; // This should be the table prefix of your UBB.threads installation.
     $conf["db"]["name"] = "sql_database"; // This should be your database which holds your UBB.threads installation.
     $conf["db"]["server"] = "localhost"; // This should be the hostname that you use to connect to MySQL.
     $conf["db"]["username"] = "sql_username"; // This should be the username to your database.

Note
You can opt to either allow the script to read out your UBB.threads configuration or connect directly to the UBB.threads database with the information supplied in this section; you can adjust this behavior by adjusting the $conf["connection"] setting (see Configuration Settings).


 

Cache & Optimization Options


     $conf["expires"]["status"] = 1; // Should we set an expires header? = 0 = No, 1 = Yes (Default: 1)
     $conf["expires"]["time"] = 86400; // In Seconds (Default: 86400)
     $conf["cache"]["directory"] = "cache/sitemap/"; // The directory you intend to use for the storage of any cache files; requires $conf["cache"]["status"] to be 1. Example: "cache/sitemap/"; directory MUST be chmodded 777.
     $conf["cache"]["status"] = 0; // Should we use the cache? (Requires the "cache_dir" value to be populated.) 0 = No, 1 = Yes (Default: 0)
     $conf["cache"]["time"] = 86400; // In Seconds, (Default: 86400)
     $conf["gzip"] = 1; // Use GZip Compression for serving pages (can add some CPU load). 0 = No, 1 = Yes (Default: 1)

Note
The GZip option will allow your server to send a compressed version of the output to the client requesting it; it should aid in speeding the delivery portion of the request. Please note that using GZip has been known to cause some minor CPU load, however most virtual hosting servers should be more than antiquate to allow this option.



 

Human Readable XML Styling


In v0.9 we added the ability to style the output of XML feeds so that a human can better read and interact with your sitemap. This is controlled with the $conf["style"]["xml"] and $conf["style"]["name"] variables within the Configuration Settings.

You can opt to use a simple CSS Stylesheet or a XSL Style which was supplied by id242; the XSL file is what we recommend using as it makes the XML output interactive. A demo of this can be found in the Demonstration section above.


 

Submissions


Not all search engines will check back automatically to re-check your sitemap; in fact, we've only noticed that Google will re-check on its own. You'll need to check back with other search engines to re-submit your sitemap if they do not auto-check it for you.

We recommend providing the following URL to search engines that support XML sitemaps:
http://www.example.com/ubb.sitemap.php?type=1&se=1&offsets=0

Those switches, as mentioned in the Usage Instructions will generate an XML sitemap with Search Engine Friendly/SEO URLs while opting not to use Offsets.


 

UBB.threads Version & URLs


URLs (and associated variables) have changed several times in the coarse of UBB.threads, below is some brief information for the $conf["version"] variable:
0 = v7.0 - v7.2.2 - Original style, non Search Engine friendly, URLs.
1 = v7.3 - v7.4.2 - Global Moderators where introduced in 7.3, which changed the guest user to user #5.
2 = v7.5 Series - v7.5 introduces SE Friendly URLs, which provides the topic title within the link itself.
3* = v7.6 Series - v7.6.0 introduces Search Engine Optimized (SEO) URLs, which provides the topic title within the link itself, and optimized the display for.

Note
There was a security issue in previous releases, which was patched in v7.5.8 which updated how URLs are written within UBB.threads; however, this patch was written poorly and introduced/allowed characters which are not valid in a URL (see RFC3986). As such, these URLs are not, and will never be, supported by this script (as all parts of this script, much like us, remain standards compliant). Users with unsupported versions (v7.5.8-v7.5.9) can opt to install id242's modification "Better URL Sanitization for SEO" (7.6.0 URLs utilize this same code)



 

Changelog


 v0.10 (October 26th, 2015)
• All queries have been cleaned up, the tally that determines a SitemapIndex's pages is done entirely server side as a single database query. Additionally the query options where combined and rewritten so that one query loads all of the needed data for the building of each Sitemap.
• Several configuration options where changed in both v0.9 and v0.10, it is recommended that you compare the options at the top of your page from your current build to the newest version that you wish to upgrade to so that you can see the changes in options and defaults.
• We now recommend everyone run as a SitemapIndex with offsets enabled as it makes your site future-proof to the amount of data you have in your Sitemap as well as break the data up into manageable chunks that should negate some server strain on large forums.
• Added support for date_default_timezone_set; you can now set the timezone for the generation times supplied by your sitemap (defaults to Pacific Standard Time for those who do not set an option).
• Support for virtual SSL over a CDN using "HTTP_X_FORWARDED_PROTO".
• Fixed a bug with caching.
• Fixed a time issue with the formatting previously used.
• Removed the recommendation of blocking user profiles in the robots.txt file as there could be some potentially valuable information as far as data is concerned.

 v0.9 (April 2nd, 2015)
• All configuration items are now stored in a $conf array.
• ASP sitemaps no longer populate both the title and description tags as they where duplicated and thus added to the bulk of sitemap pages.
• General script code cleanup, reformatting, and restructuring.
• The test string is now type 0; previously, type 0 was not utilized and would cause errors if indicated. Type 7 no longer works for any type of generation.
• We're now utilizing more current sitemap standards; previously we where utilizing earlier versions.
• There is now support for v7.6.0 style SEO URLs
• The site name parameter is no longer passed to the thread title as this just added a lot of extra bulk to the pages.
• The last reply time of a topic is now utilized in XML sitemaps.
• Utilization of an XML Stylesheet is now supported for XML Feeds; see the supplied css/sitemap.css and sitemap.xsl files for basic options.
• For compatibility for users hosting their UBB.threads install in the web root, the $conf["folder"] variable now requires a trailing slash.
• RSS Sitemaps now utilize the <ttl> tag which is fed from the $conf["cache"]["time"] variable.

 v0.8 (March 28th, 2013)
• Code Cleanup (no more Undefined variables)
• More efficient handling of SSL detection
• Preview no longer allows cache (as it should always be "live" for testing)

 v0.7 (May 26th, 2011)
• Search Engine Friendly URLs now properly match the UBB.threads' Search Engine Friendly URLs (including a more powerful SEO score for URL structure).
• You can now set the frequency that you want Search Engines to check back for new content using the $use_frequency tick.

 v0.6 (August 20th, 2009)
• This is known as the Caching and Optimization update since improvements where added to increase the speed of your install.
• Added a simple time-based caching system which will allow you to set how long the output is to be cached; this cache is built on the first visit to the script of each URL type (if a value is changed in the requesting URL it'll result in a separate caching of that output of data). The default caching time is 24 hours (this value is entered in seconds).
• Added an expires header to the code to allow client browsers (or search engines (if they support it) to cache the output of the sitemap script to their local machine for a defined period of time (default 24 hours, entered in seconds).
• Added the ability to serve the output of the script via GZip to aid in download times (basically serving a compressed dataset to the user). GZip has been known to cause some minor CPU load, but most virtual host providers should be more than capable of serving content.
• Fixed numerous cosmetic display issues and fixed a bug that's been in since ~v0.2 which made it so that the "blank" display didn't parse any data (vs falling back to the defaults you display).

Note
Some of these items may not work in your environment, hence why we added several options. You can use all of them, or none of them, or any portion of them that you wish; they're capable of running independently or in conjunction with each other.


 v0.5 (February 15, 2009)
• Removed "Priority" tag as it was optional and providing duplicate errors from Google.
• Removed some "test code" from the header of the script left in from 0.4.
• Fixed a bugglet with the "read from config" option which was added in 0.4.
• Added a "tick" to config which allows you to display the optional "changefreq" tag.
• Added the ability to pull the community name from your UBB.threads config as the feed name.
• Updated the changelog format to fix spelling and grammatical errors introduced in 0.3.

 v0.4 (May 1st, 2008)
• Version 0.4 introduces a "preview" URL set for testing connections. This preview set can be accessed as: http://www.yoursite.tld/ubb.sitemap.php?type=7&se=1
• The db connection data no longer has to be hand written into the UBB.Sitemap script; so long as you set the path to your UBB.threads Configuration directory and have "connection" set to 1, it will read the connection data from your UBB.threads configuration file.
• Additionally the Preview URL set in the "usage" area of the readme has been updated to reflect the 0.3a update. Also included in this update is compatibility with the permission update for 7.3.

 v0.3 (June 17, 2007)
• Support for SiteMapIndex's have been built in; this works well with the "offset" feature added in v0.2. The sitemap index will look at the xml sitemap data and build the proper number of "valid" links to offset pages. This is extremely useful for larger forums as they can set what level they want to offset their data (as to not go over the 50k maximum link limit with most providers) and they just need to submit one URL (versus numerous to cover additional sitemap pages.

 v0.2 (March 20th, 2007)
• This version introduces offsetting; Google has a maximum of 50k links allowed in a sitemap, some larger (huge) forums have more than 50k threads on their forums. This setting is enabled by default, and can be utilized by using the offset variable in your URLs submitted to Google/Yahoo. Additionally, it can be disabled completely in the configuration file.

 v0.1 (March 1st, 2007)
• Initial version released to the public.


 

Troubleshooting


 Invalid Date Error (1969)
This error is caused from a malformed row (or rows) in the MySQL database (either from a failed prune, topic creation, etc) which generally has blank data inserted for the row. As a timestamp of "0" in the database cannot be translated to a valid date (UBB.threads, like most other scripts stores dates in the database in the Unix Epoch format) it will be translated as a date prior to the Unix Epoch (in this case, 1969).

Note
As this error is outside of the scope of the UBB.Sitemaps script, and other items returned in the sitemap will continue being crawled, troubleshooting will require advanced (paid) support.

You can, yourself, attempt to remedy the issue, but managing the database via PHPMyAdmin or the MySQL CLI should only be attempted by advanced users as you can destroy your database; please be sure to backup your database before editing any data in the database.


You'll need to edit the _POSTS and _TOPICS tables, you'll want to sort the database by the timestamp field (post_created for the posts table, topic_created for the topics table; starting with the topics table) and look for any rows returning a "0"; you'll then want to delete the rows that return a blank set of data (generally no subject or post data) and remove those rows from the database.

After purging the rows from the database you'll want to make sure that you clear the sitemap cache (by default it is not enabled, and the default location would be "cache/sitemap/" (delete any .html files in this folder)).


 

Frequently Asked Questions


 

What does purchasing a license entitle me to?


By purchasing a license to the UBB.Sitemap script you're entitled to lifetime support of the script for the lifetime period of the script. You also receive access to a year of updates for free. After the year period; updates are purchased in blocks of 1 year for $40/yr to cover development costs. Please note that you are entitled to install this script on ONE domain per license; additional domains will require additional licenses.


 

What are the security aspects?


The UBB.Sitemap script will ignore all threads that are posted in private forums (basically, if a user whop is logged out cannot *see* a thread in a forum then the script will not publish links in that forum). Also, threads that are not approved will not show either (as they're not visible to guest users).

 

What is a Sitemap?


A sitemap will take and provide all of your current threads to search engines, thus eliminating the wait of "new discovery" of new pages on your forums.

You can read about the benefits of using sitemaps via the below quoted article from Google Webmaster Tools.

Quoting Google's Webmaster Help
What is a sitemap, and why should I have one?
A Sitemap file lets you tell us about all the pages on your site, and optionally, information about those pages, such as which are most important and how often they change. By submitting a Sitemap file, you can take control of the first part of the crawling/indexing processes: our discovery of the pages.

This may be particularly helpful if your site has dynamic content, pages that aren't easily discovered by following links, or if your site is new and has few links to it.

Sitemaps help speed up the discovery of your pages, which is an important first step in crawling and indexing your pages, but there are many other factors that influence the crawling/indexing processes. Sitemaps lets you tell us information about your pages (which ones you think are most important, how often the pages change), so you can have a voice in these subsequent steps. Other factors include how many sites link to you, if your content is unique and relevant, if we can crawl the pages successfully, and everything outlined in our webmaster guidelines.

A Sitemap provides an additional view into your site (just as your home page and HTML site map do). This program does not replace our normal methods of crawling the web. Google still searches and indexes your sites the same way it has done in the past whether or not you use this program. Sites are never penalized for using this service. This is a beta program, so we cannot make any predictions or guarantees about when or if your URLs will be crawled or added to our index. Over time, we expect both coverage and time-to-index to improve as we refine our processes and better understand webmasters' needs.

Once you have created your Sitemap, you can add it to your Google Sitemaps account and update it as your site changes.


Note
The UBB.Sitemaps script will only generate links for your UBB.threads install; it will not generate links for your entire site. We recommend generating a sitemap for your site content outside of this script versus trying to hijack built in routines, as you're only allowed a certain amount of links in a sitemap (50k) and UBB.Sitemaps adjusts itself accordingly if you're using a sitemap index with offsets.



 

Useful Links


UBB.Dev discussion for UBB.Sitemaps
Script Information on VNC Web Services
Bing Webmaster Center
Google Webmaster Tools


 

Thank You


I'd like to take the time to fully acknowledge id242 for assisting me with several tasks for this script, and for renewing my interests in everything when I was at my wit's end. Thanks to your support, hard work, and friendship, everything is looking up :).


Note
The UBB.Sitemaps script is not freeware, it is licenseware; a separate license is required for every unique instance of the sitemap script that exists on your server.

Posted on October 2nd, 2014 - Updated on January 14th, 2017
▼ Sponsored Links ▼
▲ Sponsored Links ▲

Comments

( Posted)

Related Content

▼ Sponsored Links ▼
▲ Sponsored Links ▲
For the best viewing experience please update your browser to Chrome, Firefox, or Opera